CommunityJev Benchmarks & Evaluations
Jev in Medicine: A Benchmark Evaluation (arXiv 2609.34024)
An evaluation of Jev 1.13 on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena-MCQ, and the NEJM Case Challenges, with GPT-6 Sol as reference. All 8,469 requests returned a valid answer; Jev matched GPT-6 Sol on PubMedQA but trailed on diagnostic cases.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Evaluates Jev 1.13 on four medical benchmarks, with GPT-6 Sol with and without medium reasoning as reference.
- All 8,469 requests returned a valid answer; median latency was 0.27-0.31 s and all 2,823 items cost USD 0.08.
- Jev matched GPT-6 Sol on PubMedQA (78.4% vs 78.2%) but trailed on MetaMedQA (74.8% vs 82.7%).
- On DiagnosisArena-MCQ Jev reached 59.8% versus 82.4%, and on the NEJM cases 61.8% versus 82.4%.
- On MetaMedQA Jev's expected calibration error was 0.063 versus 0.146, and its high-confidence answers were 93.4% accurate.
Watch out
arXiv preprint posted September 27, 2026 and updated October 10, 2026 (cs.AI, cs.CL, cs.LG) by Alfredo Madrid-García and Beatriz Merino-Barbancho. GPT-6 Sol was the only reference; on DiagnosisArena-MCQ Jev's probabilities discriminated poorly (AUROC 0.645 vs 0.768), and the authors state task-specific validation is required before clinical use.
Questions about this entry
Short answers drawn from the linked project or page.
MetaMedQA, PubMedQA, DiagnosisArena-MCQ, and the NEJM Case Challenges.
GPT-6 Sol, with medium reasoning and without, was the reference; the primary outcome was top-1 accuracy, with calibration, selective prediction, and recognition of unanswerable questions as key secondary outcomes.
Yes. All 8,469 requests returned a valid answer, according to the abstract.
Jev was similar to GPT-6 Sol with medium reasoning on PubMedQA (78.4% vs 78.2%), lower on MetaMedQA (74.8% vs 82.7%), and much lower on DiagnosisArena-MCQ (59.8% vs 82.4%) and the NEJM cases (61.8% vs 82.4%).
Its probabilities were the best calibrated there, with expected calibration error 0.063 versus 0.146, and answers with probability of at least 0.9 (52.9% of questions) were 93.4% accurate; GPT-6 Sol was as accurate when it accepted a similar proportion.
Of the 162 questions whose correct answer was I don't know or cannot answer, Jev chose that option for 10.5%, against 8.6% for GPT-6 Sol.
Median latency was 0.27-0.31 s, and all 2,823 items cost USD 0.08, making Jev fast and inexpensive, though the authors require task-specific validation before clinical use.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A comparative study of system-one decision models across 15 medical datasets and 8,742 instances, measuring accuracy, decision stability, and abstention. Jev rivals leading models on text-driven tasks and is weaker on numerically driven ones, while most models still fail to recognize missing answers.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark providing exact instance-level class probabilities across three simulated text domains: peer review, covert information transfer, and credit risk. Because the data-generating process is defined, exact probabilities and Bayes error are computable, enabling KL-divergence scoring of reported probabilities.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of Jev as a low-cost judge of agreement with physician-written radiology reports. A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert, and one support question per statement keeps expert agreement while using 43-45% fewer input tokens.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.