CommunityJev Benchmarks & Evaluations
General Decision Models: Benchmarking and Insights Beyond Jev (arXiv 2610.03935)
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Introduces JEVal: 11,257 bilingual instances from 36 datasets across 10 application domains.
- Evaluates 25 model configurations spanning general decision models and generative LLMs.
- Decision models are competitive when evidence resolves the decision but overstate their most likely outcome's probability.
- On τ-bench, faster local decisions cut median episode time but lower task success as errors accumulate.
- SII-Decision-27B performs on par with Jev on JEVal while answering a typical query in about 0.1 s.
Watch out
arXiv preprint posted October 2, 2026 and updated October 12, 2026 (cs.CL) by Feiyu Duan and co-authors. The proposed SII-Decision models are the authors' own, and the abstract reports benchmark results without run-to-run detail or third-party replication.
Questions about this entry
Short answers drawn from the linked project or page.
A bilingual benchmark comprising 11,257 instances from 36 datasets across 10 application domains, built to study what general decision models can reliably decide.
25 model configurations spanning general decision models and generative LLMs.
They are most competitive when decisions can be resolved from available evidence, but weaken when they require specialist knowledge or faithful uncertainty estimation.
On τ-bench, faster local decisions reduce median episode time but lower task success as decision errors accumulate over long trajectories, offsetting the fast local advantage.
They approach strong generative LLMs on individual response prediction at substantially lower inference cost, but remain weaker in user profiling with larger aggregate estimation errors and systematic bias.
Models the authors propose that internalize an open-weight LLM's own reasoning into a single-pass first-token decision through Reasoning-to-Readout Self-Distillation.
SII-Decision-27B performs on par with Jev on JEVal while answering a typical query in about 0.1 s.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A comparison of Jev-1.13.0 and two other decision models with LLMs for intent interpretation in 6G Open RAN. Jev meets the 1 s near-real-time budget on 99.8% of calls while two hosted LLMs meet it on 17.9% and 0%, with median interpretation taking 0.286 to 2.35 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A pre-registered evaluation of Jev as a memory-retirement gate deciding when a new utterance retires an old stored claim. On 500 labeled test pairs, the binary gate retires 39.2% of claims whose change is only suggested; a three-way contract cuts that to 15.5% at a 15.3-point detection cost.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev community
Posts from builders shipping with Jev right now.