CommunityJev Practices & Patterns
Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It (arXiv 2609.26758)
An evaluation of Jev and two open-weight typed decision models when option names and definitions conflict. With yes/no names, decision-flip rates are up to 70.4 percentage points higher than the 0/1 control, and mean AUC drops from 93.8% to a below-chance 23.2%.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Changes only the name-definition mapping, leaving the question, state, and the name and definition texts unchanged.
- With yes/no names, flip rates are up to 70.4 pp higher than with the 0/1 control across all four binary rules.
- Reassignment with yes/no names lowers mean AUC from 93.8% to a below-chance 23.2%.
- Random strings as option names produce flip rates close to neutral controls across all three models.
- The type-error rate stays 0% throughout, so type-correct outputs can still disobey definitions.
Watch out
arXiv preprint dated September 22, 2026 (cs.AI) by Yu Sun and co-authors, updated October 8, 2026. The evidence covers Jev and two open-weight models on 1200 constructed decision tasks; it shows option-name polarity contributes to instability, not that any single headline flip rate generalizes to production traffic.
Questions about this entry
Short answers drawn from the linked project or page.
Whether typed decision models follow explicit option definitions when option names provide a competing semantic cue, studied in Jev and two open-weight models by changing only the name-definition mapping.
On 1200 decision tasks with task-specific definitions, decision-flip rates are up to 70.4 percentage points higher with yes/no names than with the 0/1 control, a gap that holds across all four binary decision rules.
Reassignment lowered the mean AUC of the yes/no-named options from 93.8% to a below-chance 23.2%, according to the abstract's binary evaluations.
No. Random strings used as option names yield mean flip rates close to those of neutral controls across all three models, with comparable balanced accuracy before reassignment.
Flips are measured at the level of the selected definition rather than the returned name, so the analysis captures whether the decision changed meaning when only the names were remapped.
Yes. The type-error rate remains 0% throughout the experiments, which the authors present as evidence that type-correct outputs can still fail to follow explicit option definitions.
It is the competing semantic cue a name like yes or no carries relative to the definition bound to it; the paper concludes it contributes to decision instability beyond name reassignment alone.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A study of option-order sensitivity in System One models and a training-free fix called Runoff. It scores each option alone to keep a few candidates, then compares them with each candidate in every position, removing order dependence in two forward passes.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Documents a failure mode in letter-bound remote sensing classification: 97.71% OA on UCM in vocabulary but 7.35%–96.40% cross-dataset accuracy and inverted confidence (AUROC 0.38). RSAJEV replaces the letter readout with training-free prototype retrieval, reaching 89.43% on the UCM cross-dataset gate.
Jev Practices & PatternsArticle#community#research#arxiv
Community
4GitHub stars
An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Glax147/jev_ordinal_scale_biaPython
Glax147/jev_ordinal_scale_biaPython3GitHub stars
PACT turns contrastive training pairs into four loss terms for single-token typed-decision models. On a frozen 324-item holdout it matches the published recipe's accuracy (84.6% versus 85.2%) while cutting answer flips under relabelling from 13.8% to 9.8%.
Jev Practices & PatternsArticle#community#research#arxiv
A report identifying an arithmetic-dependent rejection bottleneck in Jev: on paired arithmetic problems answer-present accuracy reaches 99% while correct rejection falls to 7%. A development-selected decision threshold lifts rejection from 7% to 79% while retaining 97% answer-present accuracy.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A conceptual article that formulates a conditional Jevons hypothesis for machine evaluation: sufficiently large reductions in the marginal cost of usable machine evaluation may increase its organizational consumption where latent demand is substantial and complementary costs do not dominate.
Jev Practices & PatternsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
From the Jev community
Posts from builders shipping with Jev right now.