CommunityJev Practices & Patterns
When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev (arXiv 2609.39496)
A report identifying an arithmetic-dependent rejection bottleneck in Jev: on paired arithmetic problems answer-present accuracy reaches 99% while correct rejection falls to 7%. A development-selected decision threshold lifts rejection from 7% to 79% while retaining 97% answer-present accuracy.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- On paired arithmetic problems, answer-present accuracy reaches 99% while correct rejection falls to 7%.
- The gap persists across numerical magnitudes, operation depths, contextual formulations, and rejection labels.
- Native Boolean verification achieves 99% exact-match accuracy on the same answer-absent arithmetic cases.
- A decision threshold raises arithmetic rejection accuracy from 7% to 79% while retaining 97% answer-present accuracy.
- The failure extends to scenarios such as time calculation and capacity rounding.
Watch out
arXiv preprint posted September 30, 2026 (cs.LG, cs.CL) by Jike Zhong, Ming Li, and Yuxiang Lai. The abstract calls it a report on one failure mode, and the threshold fix is selected on separate development problems, so its 79% rejection result is a development-tuned number rather than a validated deployment setting.
Questions about this entry
Short answers drawn from the linked project or page.
Jev reliably selects correct numerical answers when available but frequently accepts incorrect alternatives when they are absent, despite an explicit other or none-of-the-above rejection option.
On paired arithmetic problems answer-present accuracy reaches 99% while correct rejection falls to 7%.
It persists across numerical magnitudes, operation depths, contextual formulations, and rejection labels, and extends to scenarios such as time calculation and capacity rounding.
TypeSafe recommends including an other or none-of-the-above option to enable rejection, but the report finds Jev frequently accepts incorrect alternatives even with that explicit rejection option present.
Yes. Native Boolean verification achieves 99% exact-match accuracy on the same answer-absent arithmetic cases, showing categorical rejection can fail even when verification succeeds.
A simple decision threshold selected on separate development problems raises arithmetic rejection accuracy from 7% to 79% while retaining 97% answer-present accuracy, without retraining or additional inference.
It is selected on separate development problems, so the improvement substantially mitigates the failure but is a development-tuned result rather than a validated deployment setting.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A theoretical study of recalibrating many related yes/no questions that each have few labels, motivated by decision models such as Jev. It derives minimax compound-risk rates and shows pooled isotonic fitting with one per-question correction achieves them, while tuning questions separately loses a factor.
Jev Practices & PatternsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A position and case-study paper on using System One decision models like Jev and the open-source Laya as decision layers in LLM-driven penetration-testing harnesses. It names four decision points, runs an exploratory 13-vulnerability NeuroSploit comparison, and proposes Rave, an unevaluated model.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Documents a failure mode in letter-bound remote sensing classification: 97.71% OA on UCM in vocabulary but 7.35%–96.40% cross-dataset accuracy and inverted confidence (AUROC 0.38). RSAJEV replaces the letter readout with training-free prototype retrieval, reaching 89.43% on the UCM cross-dataset gate.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Studies a fast learned actor in closed-loop Doom that hands control to a reasoning vision-language model only when a gate opens. On 900 held-out questions, deferring the least confident 30% of decisions gains +0.13 with doomLaya's option order, but no closed-loop variant reaches the exit.
Jev Practices & PatternsArticle#community#research#arxiv
Community
dexmac221/doomgemmaPython
dexmac221/doomgemmaPython4GitHub stars
An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Glax147/jev_ordinal_scale_biaPython
Glax147/jev_ordinal_scale_biaPythonFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.