A fast decision model is strongest when the evidence in front of it resolves the question and the answer space is clean. It weakens when the answer requires knowledge it does not have, a derivation it cannot perform in one pass, or a structure that has to hold across many steps. The published evaluations draw that boundary fairly clearly, and the boundary is more useful than a single accuracy number.
The evidence-resolvable case
The 37-dataset evaluation is the broadest map. It runs jev-1.13.0 zero-shot on 37 datasets with one frozen template per dataset, for 346,009 requests and under USD 10. It reaches 95–99% on IMDB, SST-2, HellaSwag and ARC, and 86.7% on Belebele across 122 languages. It beats Qwen3.8-27B on 27 of 37 datasets with none of Qwen's nine leads outside the bootstrap intervals, and beats Gemma-4-E4B on all 37.
The same paper reports the failure surface: all three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based quality judgments. Rotating the answer options left Jev's accuracy unchanged, and withholding the question dropped it to near chance, which rules out shallow memorization but not memorized question-answer pairs. Binary probabilities ranked well but were poorly placed relative to a fixed 0.5 threshold; thresholds tuned on training data raised micro-F1 on UNFAIR-ToS from 0.50 to 0.75.
That is the shape of the reliable case: a clean label space, evidence in the state, and a question that does not require specialist inference. The ceiling is set by the evidence and the labels, not by the model's speed.
When knowledge runs out: medicine
Jev in Medicine evaluates Jev 1.13 on MetaMedQA, PubMedQA, DiagnosisArena-MCQ and the NEJM Case Challenges, with GPT-6 Sol as reference. All 8,469 requests returned a valid answer, median latency was 0.27–0.31 s, and all 2,823 items cost USD 0.08.
| Benchmark | Jev | GPT-6 Sol |
|---|---|---|
| PubMedQA | 78.4% | 78.2% |
| MetaMedQA | 74.8% | 82.7% |
| DiagnosisArena-MCQ | 59.8% | 82.4% |
| NEJM cases | 61.8% | 82.4% |
The accuracy split is the finding. Jev matched GPT-6 Sol on PubMedQA, trailed on MetaMedQA, and fell much further on DiagnosisArena-MCQ and the NEJM cases. Its probabilities were the best calibrated on MetaMedQA, with expected calibration error 0.063 versus 0.146, and answers at probability at least 0.9, 52.9% of questions, were 93.4% accurate. On DiagnosisArena-MCQ, though, its probabilities discriminated poorly, with AUROC 0.645 versus 0.768.
Of the 162 questions whose correct answer was "I don't know" or "cannot answer," Jev chose that option for 10.5%, against 8.6% for GPT-6 Sol. Good calibration on one benchmark and weak discrimination on another are compatible. The authors require task-specific validation before clinical use, which is the right reading: the model can recognize a familiar answer without being able to work through a diagnostic case.
Structural reasoning: graphs and theory of mind
GraphDecisionBench tests whether decision models handle graph structure. It combines structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls, and evaluates Jev and related choice-based models alongside language-model baselines across fourteen configurations. Its central distinction is that accurate adjacency recognition does not guarantee broader structural correctness. Joint graph-text input did not consistently improve prediction, and feasible construction did not establish high solution quality.
Jev-ToM is a more favorable result on a different structure. It compares five typed decision models with seven generative baselines on 11,117 English questions from nine Theory of Mind benchmarks across six capability groups. Jev reached 73.2% source-balanced accuracy and ranked first or second across all six groups, leading beliefs, goals, desires and intentions, and emotions. Related-question cues accounted for most of its downstream gains. Confidence-based selection remained benchmark-dependent, and supplied reasons could repair errors or overturn correct answers. The results cover benchmark questions, not live assistant deployments.
The pattern is not "graphs bad, theory of mind good." It is that a task which can be decomposed into local judgments is a better fit than one that requires a global structural property to hold.
Long-horizon accumulation
JEVal introduces a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations. Its summary is precise: decision models are competitive when evidence resolves the decision, but weaken when they require specialist knowledge or faithful uncertainty estimation, and they overstate their most likely outcome's probability.
The long-horizon result is the one to design around. On τ-bench, faster local decisions cut median episode time but lowered task success as decision errors accumulated over long trajectories. A per-step accuracy that looks fine can compound into a poor episode. In large-scale social simulation, decision models approached strong generative LLMs on individual response prediction at substantially lower inference cost, but remained weaker in user profiling, with larger aggregate estimation errors and systematic bias.
The paper also proposes SII-Decision-4B and SII-Decision-27B, distilled models that internalize an open-weight LLM's reasoning into a single-pass first-token decision. SII-Decision-27B performs on par with Jev on JEVal while answering a typical query in about 0.1 s, according to the authors. Those are the authors' own models, and the abstract reports benchmark results without run-to-run detail or third-party replication.
Domain limits: moderation and workflow
HATEDECIDE evaluates six decision-model configurations on four hate-speech datasets against specialized, zero-shot, commercial and supervised baselines. Commercial LLMs significantly outperformed all decision models on only one of the four datasets. Supplying a dataset's definition changed up to 28% of predictions without consistently improving classification, and question decomposition significantly improved performance in only 20% of comparisons. On a diagnostic set, the best hosted decision model came within 1.6 macro-F1 points of the best commercial LLM at roughly 97% lower inference cost. The diagnostic set is small relative to the four datasets, so the gap is indicative rather than settled.
The jev-ai.org audit adds the workflow view. TypeSafe's four-workflow average is 67.8% for Jev against 74.1% for "sol," and the weakest workflow is invoice processing at 61.8% versus 79.1%. The audit treats that as a reminder to test document-heavy workflows on your own data rather than as a general verdict.
The evaluation-abundance caveat
Judgement in the Age of Jev is a conceptual piece, not an empirical result. It formulates a conditional Jevons hypothesis: sufficiently large reductions in the marginal cost of usable machine evaluation may increase organizational consumption of it where latent demand is substantial and complementary costs do not dominate. It distinguishes prediction, machine evaluation, organizational judgement and authorisation, whose costs need not fall together. It also notes that evaluations can share evidence, criteria and errors, and can scale mis-specified rubrics.
That is the right frame for cheap decision models. More evaluation is not the same as more knowledge. If the rubric is wrong, a faster model scales the error.
What to do
- Test items just beyond the observed knowledge boundary, not only in-distribution cases.
- Pair the model with retrieval when the answer depends on facts it may not hold.
- Decompose multi-step judgments into questions the model can answer locally, then combine in code.
- Verify derived answers with a second pass or a code check.
- Do not use the model for counting or date arithmetic; the jaggedness notes explain why.
- Measure calibration conditional on correctness, not only overall.
- Evaluate the whole trajectory, not just the per-step accuracy.
The design pattern is the one in designing around jagged edges. Measuring Jev covers the field reports, and what the papers found covers the broader literature. The System Two task, jaggedness and calibration entries define the terms.
