CommunityJev Benchmarks & Evaluations
Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation (arXiv 2609.33843)
An independent reproduction of the shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, finding it uniformly under-confident rather than over-confident. A single fitted temperature removes most miscalibration, while the preregistered isotonic selection failed its held-out contrast.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Signed confidence-accuracy gap of -0.214: every occupied reliability bin is more accurate than confident.
- Sign uniformity collapses every binned ECE variant to the same value, 0.214.
- A disjointly fitted temperature of 0.469 cuts held-out ECE from 0.204 to 0.037.
- Re-running the checkpoint reproduced the card's headline accuracy: 0.767 vs 0.766.
- The frozen gate beat random escalation but missed its 10% accepted-set error target on both tracks.
Watch out
arXiv preprint posted September 27, 2026 (cs.AI, cs.CL, cs.LG) by Gowthamkumar Nandakishore; 31 pages, 8 figures, with frozen preregistration and run manifests in ancillary files. The author states no affiliation with the model's publisher, the dataset's publisher, or TypeSafe, and every score measures agreement with a synthetic teacher.
Questions about this entry
Short answers drawn from the linked project or page.
The shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, is uniformly under-confident. The signed confidence-accuracy gap is -0.214, and every occupied reliability bin's accuracy exceeds its confidence.
The model card frames the risk as over-confidence, but the measured direction is the opposite, and the direction decides which way a confidence-gated cascade fails, so the correction differs from what the card anticipates.
A single disjointly fitted temperature of 0.469, which sharpens the scores and removes most of the miscalibration, cutting held-out ECE from 0.204 to 0.037 and outperforming the shipped per-option-count table.
The frozen selection rule chose isotonic regression, which overfit and failed its held-out NLL contrast on both tracks, so hypothesis H2 was not supported by the preregistered analysis.
Yes. Re-running the released checkpoint on its full official test split reproduced the card's headline accuracy: 0.767 against the card's 0.766, supporting the reproduction before the further preregistered tests.
The retrospective E1 reproduction preceded the analysis freeze; E2 through E8 were prospectively preregistered, and 20 of 22 executed confirmatory tests reject under Benjamini-Hochberg FDR at q=0.05, with two descoped.
Every score measures agreement with a synthetic teacher whose self-agreement ceiling of 0.735 the specialist exceeds, the frozen gate missed its 10% accepted-set error target on both tracks, and an exploratory out-of-distribution probe found no zero-shot transfer at 0.617 accuracy.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsHacker News: A one-token logprob wrapper that also classifies webcam frames.
allanrbo151 pointsA single function Jev-like wrapper for LLMs, including vision modelsRead the thread on Hacker NewsRelated terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
From the Jev guides
Original write-ups on what this entry is for.
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
Training recipes, looped architectures, and language ports: what the open side of the typed-decision class shipped this month, and how to pick by runtime and calibration instead of the leaderboard.
Jev Models & Reimplementations#open-models#training#local
Read article
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.