CommunityJev Benchmarks & Evaluations
Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding (arXiv 2609.27678)
An evaluation of Jev and nine language models on ContractNLI with repeated requests that vary visible hypotheses, requested answers, and output order. Corrections and regressions can cancel so accuracy stays unchanged while decisions change; Jev is cheapest and fastest for the joint baseline.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Evaluates Jev and nine language models on ContractNLI with repeated requests.
- Requests preserve the contract and target while varying hypotheses, requested answers, and output order.
- Corrections and regressions can cancel, leaving unchanged accuracy despite changed decisions.
- Majority voting over recorded responses yields small, uncertain gains, with repeated wrong answers accounting for much error.
- Jev is cheapest and fastest for the joint baseline, while hosted language models achieve higher accuracy.
Watch out
arXiv preprint posted September 23, 2026 and updated October 10, 2026 (cs.CL) by Fan Zhang and co-authors. A four-model follow-up adds an equal-call repetition control and coverage of remaining test contracts, but the finding that consistent answers can preserve errors means aggregate accuracy should not be read as decision stability.
Questions about this entry
Short answers drawn from the linked project or page.
Which judgments remain correct when requests change: repeated requests preserve the contract and target while varying visible hypotheses, requested answers, and output order.
Corrections and regressions can cancel, leaving unchanged accuracy despite changed decisions, and consistent answers can also preserve errors.
Majority voting over recorded responses yields small, uncertain gains, with repeated wrong answers accounting for much of the observed error.
An equal-call repetition control and coverage of the remaining test contracts.
Different error profiles behind aggregate model rankings, so similar-looking scores can come from different kinds of mistakes.
Jev is cheapest and fastest for the joint baseline, while hosted language models achieve higher accuracy.
Whether additional calls improve correctness and what the resulting decisions cost.
More Jev resources like this
0GitHub stars
Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
2GitHub stars
An automated framework that traces adversarial context through Jev-backed program decisions to downstream effects. Across 228 real-world Jev applications and 2,273 test cases, 81.5% show decision flips, 98.6% with potential security impact, and Decision Flip Rate beats the strongest baseline by 50.4 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
xiaoxiaohg8d/JevProbe-Python
xiaoxiaohg8d/JevProbe-PythonA controlled evaluation of Jev as a source-support verifier for GPT-4.1-mini calculation traces on financial reports. Holding operands and arithmetic fixed while swapping citations between same-number cells reveals wrong-role citations that pass and valid alternative citations that are withheld.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
David-DL-Space/sys1-evalJavaScript
David-DL-Space/sys1-evalJavaScriptA benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonA pre-registered study of whether jev-1.13.0 satisfies independence of irrelevant alternatives, using 299,463 calls on intent and topic classification. Every added option shifts the log-odds between the gold and strongest wrong label by 0.20–0.52 nats RMS, 1.2 to 3 times run-to-run variation.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.