CommunityJev Benchmarks & Evaluations
TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models (arXiv 2610.11392)
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Built from seven policy-labelled generators and nine evaluation suites for typed decision models.
- Reports accuracy as median and range across paraphrases, plus calibration against a finite-sample noise floor.
- The hosted model follows the stated policy but is wording-sensitive and systematically underconfident.
- Under asymmetric costs, using the hosted model's probabilities can be worse than taking its top answer.
- Evaluates a hosted model, an open encoder, and open decoders spanning 0.8B–9B parameters on identical items.
Watch out
arXiv preprint posted October 8, 2026 and updated October 9, 2026 (cs.AI, stat.AP) by Rahul Sharma, Andrew B. Duncan, Gaétan Marceau Caron, and Sebastian J. Vollmer. The hosted-model findings describe one unnamed service, and the abstract does not report its cost per item.
Questions about this entry
Short answers drawn from the linked project or page.
Typed decision models that output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes through a non-generative interface.
From seven policy-labelled generators and nine evaluation suites, with accuracy reported as median and range across paraphrases.
Relative to the finite-sample noise floor of a matched, perfectly calibrated predictor, rather than against an absolute target.
It follows the stated policy but is wording-sensitive and systematically underconfident, which matters because software acts on these probabilities through thresholds and escalation rules.
Using the hosted model's probabilities can be worse than taking its top answer when costs are asymmetric, the authors report.
Decoders route exactly but are slow as options or questions are added, and the decoder is least accurate on policy questions and degrades with more options.
Policy adherence, wording robustness, probability quality, and induced decision outcomes, since miscalibrated or wording-sensitive probabilities can trigger unintended actions with no textual rationale to inspect.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: 100,000 viral posts in 20.4 seconds
X: SEO and GEO fixes, 90% cheaper
X: Hard-coded rules moved to Jev
YouTube: Joe Maddalone tries Jev as a routing and filtering step, with a transcript-analyzer demo.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An independent reproduction of the shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, finding it uniformly under-confident rather than over-confident. A single fitted temperature removes most miscalibration, while the preregistered isotonic selection failed its held-out contrast.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.