CommunityJev Benchmarks & Evaluations
Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration (arXiv 2609.35342)
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Each item's exact probability P(A) is known by construction and queried via Noul, Choice, and Score.
- Items are evaluated by total variation distance from the ground-truth distribution.
- The author hypothesizes that Jev suppresses a third truth value beyond True and False.
- In Choice answers P(A) and P(¬A) are presented as if they sum to one, while a term P(U) ≠ 0 is missing.
- Recovering P(U) improves median soft accuracy on Choice answers from 0.771 to 0.978.
Watch out
arXiv preprint posted September 28, 2026 (cs.AI, cs.LO, eess.SY) by Riccardo Porcedda. The third-value explanation is an inference from observed Score and Choice behavior on constructed items rather than a confirmed property of the hosted service, and the abstract compares only Jev and the open-source SemIf baseline.
Questions about this entry
Short answers drawn from the linked project or page.
A dataset of True/False questions about a proposition A for which the exact probability P(A) is known by construction, intended to test whether every returned option probability has the right numerical meaning.
Jev's central promise is that its probabilities are calibrated, but the author notes the claim is not backed by a public test and existing benchmarks evaluate confidence calibration rather than numerical meaning.
Each item is queried through Jev's three primitives, Noul, Choice, and Score, and evaluated by total variation distance from the ground-truth distribution, which estimates a soft accuracy.
A pattern that can be explained by assuming Jev suppresses a third truth value beyond True and False.
In Choice answers, P(A) and P(¬A) are presented as if P(A) + P(¬A) = 1, while a term P(U) ≠ 0 is missing from the sum.
Recovering P(U) improves median soft accuracy in Choice answers from 0.771 to 0.978.
SemIf, an open-source Choice-style baseline, is evaluated on Sys1Cal-v1, though the paper focuses more deeply on Jev's Score and Choice calibration.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
4GitHub stars
An audit of whether Jev's confidence reflects missing knowledge rather than chance, using over 15 public datasets and 6 generated task families with paired interventions. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An independent reproduction of the shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, finding it uniformly under-confident rather than over-confident. A single fitted temperature removes most miscalibration, while the preregistered isotonic selection failed its held-out contrast.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.