Healthcare is where a wrong label is most expensive, so the useful question is not whether a model can answer a medical question. It is which administrative judgment can be bounded tightly enough that a person can review the exceptions. Triage routing, note extraction and factuality checks fit that test far better than diagnosis.
Triage is routing, not diagnosis
The Intake Router routes an intake message to a care pathway and flags same-day urgency in one request. Its answer space is deliberately administrative: no question asks for a condition or a dosage. Every same-day flag goes to a triage nurse, every low-confidence case goes to a person, and the repository ships intake messages with embedded instructions so prompt injection can be tested before routing.
That is the correct scope. A routing decision can be evaluated by whether the right queue received the message and whether the urgent cases were flagged, not by whether the model "understood" the illness. The README is also explicit that this is an administrative aid, not a medical device and not validated for any clinical use. The human review path is part of the product, not a fallback bolted on afterward.
Documentation is extraction with confidence
MedJev turns free-text clinical notes into table-ready data by answering predefined typed questions: Noul for yes/no with a probability, Choice for one of 2 to 255 named options, and Score for an ordered level, with a confidence attached to every answer. Its README claims thousands of notes an hour on one consumer GPU, no subscription or per-use fees, and that patient text never leaves the network. It also notes an optional hosted Jev baseline.
Those throughput and accuracy claims are the project's own; the captured README lists no benchmarks, deployed results or licence text beyond Apache-2.0 metadata. The more useful design detail is the review split: clear-cut charts can be accepted automatically and doubtful ones sent to a person. Adding a new field needs a few hundred labeled example charts and a few hours of retraining, per the comparison table, which is a realistic maintenance estimate rather than a promise of zero-shot extraction.
The same pattern appears in Jev-MedQA, which selects answers by reading Qwen3.5-4B option scores in one forward pass with no additional training. It reports 69.42% versus 67.41% accuracy across 153,889 questions from nine medical QA sets, and 0.182 seconds per question versus 1.891 seconds for generation. That is a Jev-style paradigm implementation built from SemIf, not TypeSafe's hosted model, and its measurements used RTX 3090 GPUs with thinking mode disabled.
Benchmark results are not clinical validation
The Jev in Medicine evaluation is an independent preprint, and it is careful about what it measured. All 8,469 requests returned a valid answer, median latency was 0.27 to 0.31 seconds, and all 2,823 items cost USD 0.08. Jev matched GPT-6 Sol on PubMedQA at 78.4% versus 78.2%, but trailed on MetaMedQA at 74.8% versus 82.7%, on DiagnosisArena-MCQ at 59.8% versus 82.4%, and on the NEJM cases at 61.8% versus 82.4%.
The calibration result is more favorable: on MetaMedQA Jev's expected calibration error was 0.063 versus 0.146, and its answers at 0.9 probability or higher, 52.9% of questions, were 93.4% accurate. On the 162 questions whose correct answer was "I don't know" or "cannot answer," Jev chose that option 10.5% of the time versus 8.6% for GPT-6 Sol. The authors state that task-specific validation is required before clinical use, and GPT-6 Sol was the only reference.
The reliability study broadens the picture to 15 datasets and 8,742 instances. Jev rivals leading models on text-driven tasks but is less effective on numerically driven ones, and under question-format variations it has the lowest decision reversal rate and small accuracy swings. Among the open alternatives, only StartLux shows comparably strong robustness. Most evaluated models still struggle to recognize missing correct answers or abstain, so robustness does not equal readiness.
Factuality checks are the strongest use
Judging whether a generated report is supported by a source is a narrower task than producing the report, and the radiology judge study measures it directly. Jev checks whether each statement in an AI-generated radiology report is supported by a physician-written reference, combining judgments in both directions to catch unsupported claims and omissions.
A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert. One support question per statement retains similar expert agreement to a seven-question configuration with 43 to 45% fewer judgment input tokens, and judgments cost under three cents per hundred report pairs at the documented API price. In a controlled-error test Jev detects false negation with an AUROC of 0.977. The limit is equally clear: local RadMatch achieves stronger agreement on clinically significant errors in both expert datasets, and benchmark agreement reflects report size and error definitions as well as medical error detection.
Calibration and hallucination benchmarks
The MedHallu benchmark is a pre-registered run with every run file published. Jev scored 92.9% on the 1,000-item test split at a 204 ms median and USD 0.03 per 1,000 checks, 2.6 to 6.5 times faster and 5 to 35 times cheaper than four LLMs. On the 37% of items where Jev was at least 90% confident, it was 99.5% accurate with an AUROC of 0.974, and letting Jev settle those items kept each LLM's accuracy with 37% fewer LLM calls.
Two caveats matter. MedHallu still scores 66 to 83% with the abstract removed, so it does not only test source checking, and Jev's confidence numbers are not literal odds. OmniMed-Jev adds a multimodal result: calibration error falls by up to an order of magnitude and reliability error by up to two times against a matched generative baseline, while counting is the one family where the generative baseline stays ahead. Its authors also say the results are not evidence of clinical readiness.
What a safe deployment looks like
A safe clinical pipeline separates three layers. The model answers bounded questions; code owns the thresholds, the audit log and the escalation; a person owns every consequential action. The selective decision pattern is the formal version: return an answer only when confidence clears a floor, otherwise abstain and route.
The healthcare papers roundup and calibration beyond ECE cover the measurement side. The confidence gates in production piece covers the operational side: thresholds are per action and do not transfer across question types. TypeSafe's legal page adds the data commitment, no training on user data and zero retention for enterprise through sales, though that is enterprise-gated and the page is a thin index.
None of this makes a model a clinician. It makes the model's answer one input to a workflow that can be inspected, overridden and audited, which is the only version of this that belongs near a patient.
