CommunityJev Benchmarks & Evaluations
Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev (arXiv 2610.16655)
A pre-registered study of whether jev-1.13.0 satisfies independence of irrelevant alternatives, using 299,463 calls on intent and topic classification. Every added option shifts the log-odds between the gold and strongest wrong label by 0.20–0.52 nats RMS, 1.2 to 3 times run-to-run variation.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Uses 299,463 calls to test jev-1.13.0 on BANKING77, CLINC150, and AG News.
- Every added option moves the gold-versus-strongest-wrong log-odds; RMS shift is 0.20–0.52 nats.
- Shifts are directional: similar options draw probability from the rival, paraphrases split less than substitution implies.
- Distortion grows with log option count, reaching about 1 nat at 255 options; off-domain padding hurts the gold label.
- Scoring each option with an independent yes/no question removes option-set dependence but picks the right answer 8.7 points less often.
Watch out
arXiv preprint posted October 12, 2026 (cs.AI, cs.CL) by Dushyant Rajput, Nirdesh Chauhan, and Siddharth Kosaraju. The study covers jev-1.13.0 on intent and topic classification only; the authors place it between two local baselines, and conclusions about other decision models or tasks are not established by these calls.
Questions about this entry
Short answers drawn from the linked project or page.
Independence of irrelevant alternatives: that adding an option which is neither the right answer nor its strongest rival should leave the odds between those two unchanged.
299,463 calls on intent classification with BANKING77 and CLINC150 and topic classification with AG News, all against jev-1.13.0.
Every added option moves the log-odds between the gold label and the strongest wrong label, with a root-mean-square systematic shift of 0.20–0.52 nats, 1.2 to 3 times the run-to-run variation.
Yes. An option similar to the wrong answer draws probability from it, shifting +0.10 nats toward the gold label, while a paraphrase of the wrong answer yields an intermediate pattern.
It grows steadily with the logarithm of the number of options, reaching about 1 nat at 255 options, and off-domain padding pushes the odds against the gold label.
Scoring each option with an independent yes/no question removes the option-set dependence but picks the right answer 8.7 points less often than the hosted model.
The authors conclude they are best read as relative to the option list they were computed over, since questions packed into one call do not interact (bias below 0.001).
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonA pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of Jev on two stance detection datasets, VAST for English texts and ZS-CSD for Chinese conversations, against four general-purpose LLMs and two fine-tuned models. Jev matches GPT-5.6 on VAST but falls behind stronger LLMs on ZS-CSD.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A first empirical study of Jev on application classification for network flows, using ten packet-header features from CESNET-QUICEXT-25. Forty labeled examples raise accuracy from 9.80% to 28.42%, while tree ensembles trained on 8,000 records reach about 67-70%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.