CommunityJev Benchmarks & Evaluations
Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality (arXiv 2609.27607)
An evaluation of Jev as a low-cost judge of agreement with physician-written radiology reports. A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert, and one support question per statement keeps expert agreement while using 43-45% fewer input tokens.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Checks whether each statement is supported by the other report and combines judgments in both directions.
- A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert.
- One support question per statement retains similar expert agreement to seven with 43-45% fewer judgment input tokens.
- Judgments cost under three cents per hundred report pairs at the documented API price.
- In a controlled-error test Jev detects false negation with an AUROC of 0.977.
Watch out
arXiv preprint posted September 23, 2026 (cs.CL, cs.AI) by Jiaju Huang and co-authors. Local RadMatch achieves stronger agreement on clinically significant errors in both expert datasets, and finding-count and error-scope analyses show benchmark agreement reflects report size and error definitions as well as medical error detection.
Questions about this entry
Short answers drawn from the linked project or page.
Whether an AI-generated radiology report's statements are supported by a physician-written reference, checking each statement in both directions to capture unsupported claims and omissions.
Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert with expert error counts.
It outperforms an open natural language inference judge under matched decomposition and aggregation.
One support question per statement retains similar expert agreement to a seven-question configuration while using 43-45% fewer judgment input tokens.
At the documented API price, judgments cost under three cents per hundred report pairs, excluding local decomposition.
Jev detects false negation with an AUROC of 0.977 in a separate controlled-error test.
Local RadMatch achieves stronger agreement on clinically significant errors in both expert datasets and on total errors in the shared RadEvalExpert subset, and benchmark agreement reflects report size and error definitions as well as medical error detection.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Reddit: Localization QA with a decision model: meaning, terminology, and placeholders as separate checks.
Hacker News: Pick Verify: a single decision at the warehouse pick station catches confusable picks.
Show HN: Verifying robot picks with a decision modelRead the thread on Hacker NewsMore Jev resources like this
A comparative study of system-one decision models across 15 medical datasets and 8,742 instances, measuring accuracy, decision stability, and abstention. Jev rivals leading models on text-driven tasks and is weaker on numerically driven ones, while most models still fail to recognize missing answers.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A controlled evaluation of Jev as a source-support verifier for GPT-4.1-mini calculation traces on financial reports. Holding operands and arithmetic fixed while swapping citations between same-number cells reveals wrong-role citations that pass and valid alternative citations that are withheld.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An evaluation of Jev 1.13 on four medical benchmarks: MetaMedQA, PubMedQA, DiagnosisArena-MCQ, and the NEJM Case Challenges, with GPT-6 Sol as reference. All 8,469 requests returned a valid answer; Jev matched GPT-6 Sol on PubMedQA but trailed on diagnostic cases.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
From the Jev community
Posts from builders shipping with Jev right now.