CommunityJev Benchmarks & Evaluations
How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs (arXiv 2610.18529)
A comparative study of system-one decision models across 15 medical datasets and 8,742 instances, measuring accuracy, decision stability, and abstention. Jev rivals leading models on text-driven tasks and is weaker on numerically driven ones, while most models still fail to recognize missing answers.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Covers 15 medical datasets and 8,742 instances under a single evaluation protocol.
- Jev rivals leading models on text-driven tasks but is less effective on numerically driven ones.
- Under question-format variations Jev has the lowest decision reversal rate and small accuracy swings.
- Among evaluated open-source alternatives, only StartLux shows comparably strong robustness.
- Normalizing option-wise binary scores retains more accuracy than choice scoring.
Watch out
arXiv preprint posted October 13, 2026 (cs.CL) by Yanrui Du and eight co-authors. The study is an evaluation under its own protocol, not a clinical validation; most evaluated models still struggle to recognize missing correct answers or abstain, so the results give early guidance rather than evidence of medical readiness.
Questions about this entry
Short answers drawn from the linked project or page.
The study compares system-one decision models across 15 medical datasets comprising 8,742 instances under a unified protocol, and assesses decision stability and the ability to recognize unreliable decisions alongside accuracy.
Jev rivals leading models on primarily text-driven tasks but is less effective on numerically driven tasks, which the authors describe as a distinct capability boundary for system-one decision models in medicine.
It shows strong robustness: the lowest decision reversal rate among the evaluated models and relatively small accuracy fluctuations when the question format changes.
Among the evaluated open-source alternatives, only StartLux exhibits comparable robustness, according to the abstract.
Most evaluated models show limitations in both recognizing missing correct answers and using an optional abstention choice, so the robustness strengths do not extend to reliable abstention awareness.
Normalizing option-wise binary scores yields higher retained accuracy than choice scoring, showing that changing how decisions are scored can improve Jev's answer selection.
The findings provide early guidance for improving and applying decision models in medicine, but they come from one comparative protocol and should not be read as a clinical readiness claim.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
An evaluation of Jev 1.13 on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena-MCQ, and the NEJM Case Challenges, with GPT-6 Sol as reference. All 8,469 requests returned a valid answer; Jev matched GPT-6 Sol on PubMedQA but trailed on diagnostic cases.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
4GitHub stars
An audit of whether Jev's confidence reflects missing knowledge rather than chance, using over 15 public datasets and 6 generated task families with paired interventions. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of Jev as a low-cost judge of agreement with physician-written radiology reports. A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert, and one support question per statement keeps expert agreement while using 43-45% fewer input tokens.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.