CommunityJev Benchmarks & Evaluations
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure (arXiv 2609.26550)
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Compared against sixteen generative and reward-model judges with blinded human adjudication.
- JEV stays within three points of GPT-6 where the verdict can be read off the text, at 0.36% of its fee.
- It falls behind where the verdict must be derived, as in math, code, and logic.
- A pre-frozen threshold made the accept-or-escalate cascade 0.9 points more accurate than GPT-6 on 1,610 held-out pairs.
- The cascade cost 41% of GPT-6's fee and matched its accuracy exactly in a pre-specified live test.
Watch out
arXiv preprint posted September 22, 2026 and updated October 9, 2026 (cs.AI) by Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. GPT-6 is the comparison baseline, the threshold was frozen in advance rather than chosen per workload, and confidence routing weakens on style-adversarial pairs and reference-free prose.
Questions about this entry
Short answers drawn from the linked project or page.
An evaluation setup that uses JEV, a decision-only judge returning label probabilities instead of generated text. Its confidence decides whether to accept the verdict or escalate the case to a reasoning judge, targeting the cost and latency of reasoning judges.
Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a median latency of 0.15 seconds.
Where the verdict must be derived rather than read off, as in math, code, and logic. Confidence marks this boundary, which is what makes escalation viable rather than a blanket replacement for reasoning judges.
With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs, at 41% of its fee. In a pre-specified live test on two new workloads the cascade matched GPT-6's accuracy exactly.
On style-adversarial pairs and reference-free prose, where the confidence signal becomes less reliable as a guide to correctness. The paper closes with a simple recipe for validating thresholds locally instead of trusting a global setting.
Reasoning judges are slow and costly, and most evaluation items do not need one: JEV returns label probabilities in a single decision, and only the cases its confidence marks as hard go on to a reasoning model.
It is an arXiv preprint, initially posted September 22, 2026 and updated October 9, 2026; the update's comment notes that a comparison against an open-source alternative to Jev was added to appendix A, and the main claims are the authors' own evaluation.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
YouTube: Jev explained for routing and judging, with a comparison against LLMs.
Hacker News: Vercel on using Jev for defined decisions and GPT-6 Astra when the job also has to write or use tools.
flashbrew4 pointsTypeSafe AI Jev vs. GPT-6 AstraRead the thread on Hacker NewsHacker News: ASTGL replay chose local Laya routing over hosted Jev on one Mac Studio setup.
Jmeg8r15 pointsDoes an open-weight decision model beat a hosted one? Jev vs. LayaRead the thread on Hacker NewsRelated terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An independent reproduction of the shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, finding it uniformly under-confident rather than over-confident. A single fitted temperature removes most miscalibration, while the preregistered isotonic selection failed its held-out contrast.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
From the Jev community
Posts from builders shipping with Jev right now.