CommunityJev Benchmarks & Evaluations
Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge (arXiv 2610.08675)
A controlled evaluation of Jev as a source-support verifier for GPT-4.1-mini calculation traces on financial reports. Holding operands and arithmetic fixed while swapping citations between same-number cells reveals wrong-role citations that pass and valid alternative citations that are withheld.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Uses Jev as a source-support verifier for GPT-4.1-mini calculation traces on financial reports.
- Holds operands and arithmetic fixed while moving citations between same-number cells.
- A signed-number-at-pointer baseline explains most recovery over exact quotation checks.
- Explicit column labels improve selected wrong-role decisions while lowering support for some equivalent evidence.
- A follow-up on 36 new source pages, labeled by a non-author reviewer, exposes the same tradeoff.
Watch out
arXiv preprint posted October 6, 2026 (cs.CL) by Chuhong Xu and five co-authors. The follow-up review covers 36 new source pages labeled by a single non-author reviewer, so the tradeoff between detecting role errors and retaining valid citations is measured in a controlled setting rather than a deployed financial assistant.
Questions about this entry
Short answers drawn from the linked project or page.
Financial reports repeat values across periods, metrics, and accounting lines, so an LLM-generated calculation can be numerically correct while citing the wrong financial role; the paper tests what probabilistic evidence verification adds beyond number matching.
Operands and arithmetic are held fixed while citations are moved between same-number cells, with controls that express equivalent facts, so any change in verdict isolates the citation's role.
It explains most recovery over exact quotation checks, according to the abstract, leaving a smaller role-recognition problem for probabilistic verification to address.
Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence, a tradeoff the authors highlight.
A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends the evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations.
Both wrong-role citations that pass verification and valid alternative citations that are withheld, which number matching alone cannot separate.
A controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed, making numerical correctness, cited-role support, and acceptance outcomes separately assessable.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Reddit: Localization QA with a decision model: meaning, terminology, and placeholders as separate checks.
Hacker News: Pick Verify: a single decision at the warehouse pick station catches confusable picks.
Show HN: Verifying robot picks with a decision modelRead the thread on Hacker NewsMore Jev resources like this
An evaluation of Jev and nine language models on ContractNLI with repeated requests that vary visible hypotheses, requested answers, and output order. Corrections and regressions can cancel so accuracy stays unchanged while decisions change; Jev is cheapest and fastest for the joint baseline.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An evaluation of Jev as a low-cost judge of agreement with physician-written radiology reports. A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert, and one support question per statement keeps expert agreement while using 43-45% fewer input tokens.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
2GitHub stars
An automated framework that traces adversarial context through Jev-backed program decisions to downstream effects. Across 228 real-world Jev applications and 2,273 test cases, 81.5% show decision flips, 98.6% with potential security impact, and Decision Flip Rate beats the strongest baseline by 50.4 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
xiaoxiaohg8d/JevProbe-Python
xiaoxiaohg8d/JevProbe-PythonAn audit of request independence in seven open Jev-style checkpoints, using 276 three-passage states built from 92 TREC retrieval queries. Paired interventions show request expansion flips 9.1%/8.0% of Nimble's judgments and changes up to 21.4% of selection-or-abstention actions.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev community
Posts from builders shipping with Jev right now.