User repository on GitHub
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models (arXiv 2609.38827)
An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.
- Stars
- 4
- Language
- Python
- License
- MIT
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- JEV assigns 38.8% of predictions and 51.3% of errors to Neutral on ANLI despite 74.95% accuracy and balanced gold labels.
- Across 36 ordinal datasets, decisions use 67-76% of effective gold support versus 87-102% on four nominal tasks.
- Scales refined from K=2 to K=14 show utilization falling to 26-75% at K=14 for every model.
- Randomizing candidate order weakens but does not remove the compression.
- BA-LoRA post-training raises gold-relative utilization from roughly 47% to 86% on eight supervised scales at both KEV sizes.
Watch out
arXiv preprint dated September 30, 2026 (cs.AI) by Tianxiang Gao and co-authors. The study covers JEV 1.13 and three open KEV models; utilization gaps are measured against gold labels rather than task outcomes, and the effect on end-task accuracy is not established.
Questions about this entry
Short answers drawn from the linked project or page.
It is the paper's name for decision-stage candidate-space compression: direct-decision models failing to use the full ordinal scale supplied by the user, distinct from accuracy, gold imbalance, fixed position, and candidate count alone.
JEV assigns 38.8% of all predictions and 51.3% of errors to Neutral despite 74.95% accuracy, nearly balanced gold labels, and balanced candidate positions.
Across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support, compared with 87-102% on four nominal tasks in the same investigation.
No. Randomizing candidate order weakens but does not remove the compression, which the authors say points to something beyond fixed candidate positions.
Holding items and source scores fixed while balancing gold support and positions, the authors refine scales from K=2 to K=14; utilization falls for every model and reaches 26-75% at K=14, though candidate probabilities remain broad for most.
Yes, apparently. Targeted BA-LoRA post-training raises gold-relative utilization from roughly 47% to 86% on eight supervised scales at both KEV sizes, showing the compression is learned and modifiable rather than an immutable architectural limit.
Anyone relying on low-latency structured labels and scores for classification and automatic evaluation, since the paper argues reliability requires faithful use of the ordinal scale, not just accuracy.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
3GitHub stars
PACT turns contrastive training pairs into four loss terms for single-token typed-decision models. On a frozen 324-item holdout it matches the published recipe's accuracy (84.6% versus 85.2%) while cutting answer flips under relabelling from 13.8% to 9.8%.
Jev Practices & PatternsArticle#community#research#arxiv
A theoretical study of recalibrating many related yes/no questions that each have few labels, motivated by decision models such as Jev. It derives minimax compound-risk rates and shows pooled isotonic fitting with one per-question correction achieves them, while tuning questions separately loses a factor.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A training-free framework with two implementations that turn emotion classification into Jev judgments: one decomposes tasks into atomic judgments, the other aggregates complementary judgment paths. Across eight datasets standard Jev averages 62.93% macro-F1 versus 67.28% for the strongest LLM baseline.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A study of option-order sensitivity in System One models and a training-free fix called Runoff. It scores each option alone to keep a few candidates, then compares them with each candidate in every position, removing order dependence in two forward passes.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Shows that open instruction-tuned models already read decisions from answer-letter logits without training, and that LoRA fine-tuning with a decision head lifts them past Jev: Gemma 4 31B goes from 79.6 to 84.4 on the new 2,352-question OpenDecisions set and scores 81.5 on JevBench v1.7.
Jev Models & ReimplementationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
From the Jev community
Posts from builders shipping with Jev right now.