Article
CommunityJev Benchmarks & Evaluations
To Jev or Not to Jev: When Can System-1 Decision Models Replace LLM Judges?
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Evaluates 14 System-1 decision models against four generative judges on eleven established judge benchmarks.
- Five of the 14 match GPT-5.6 without reasoning on all five pairwise and best-of-4 benchmarks.
- With a reference answer, eleven models reach 90.3-93.4% on JudgeBench.
- On PPE best-of-16, reference-guided System-1 judges match GPT-5.6 with reasoning at roughly one-sixteenth the cost per prompt.
- An appended note to the evaluator reverses 0-5% of correct verdicts for five models but 17-49% for the other nine.
Questions about this entry
Short answers drawn from the linked project or page.
Fourteen System-1 decision models, selected with a pre-stated protocol, were compared against four generative judges: Claude Haiku 5.5, GPT-5.6 with and without reasoning, and Claude Opus 5.5 with thinking.
The evaluation covers eleven established judge benchmarks, including JudgeBench, LLMBar, RewardBench 2, and PPE best-of-16.
The two strongest, DReX-Max and d1-ultra, are significantly more accurate than Haiku on JudgeBench, LLMBar and RewardBench 2.
Conditioned on a reference answer generated once per prompt, eleven models reach 90.3-93.4% on JudgeBench, statistically indistinguishable from Opus 5.5 and GPT-5.6 with reasoning.
On PPE best-of-16, every reference-guided System-1 judge matches GPT-5.6 with reasoning at roughly one-sixteenth of its cost per prompt.
Robustness is a property of the individual model rather than the System-1 design: an appended note to the evaluator reverses 0-5% of correct verdicts for five models but 17-49% for the other nine.
Two general System-1 leaderboards that contain no judging tasks predict judging accuracy with a Pearson correlation of 0.95-0.96.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Agent Handoff Gate: agents verify what they hand off
X: Jev as an LLM guardrail inside Agentgateway
X: Vercel's fx safety reviewer, measured at 18× faster
Reddit: Early experiments wiring Jev into Pi as a safety layer and model router.
More Jev resources like this
arXiv paper presenting the first systematic bias and fairness audit of Jev, a typed-decision model for recruitment. It measures how demographic attributes and candidate cues shift screening scores, rankings, and shortlisting, against a noise floor, plus validity, latency, and cost.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#research
Community
arXiv paper presenting Slot Attack, a black-box attack that appends text to evidence to steer decision models toward an attacker-chosen wrong option. Evaluates seven decision models on 16 Decision Bench tasks across seven domains.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.