CommunityJev Benchmarks & Evaluations
Typed Decision Models: An Early Evidence Audit and Evaluation Checklist (arXiv 2609.32160)
An audit of 28 papers posted between 19 and 24 September 2026, nine days after TypeSafe released Jev, relating their findings to earlier work and deriving a 14-item evaluation checklist. The typed readout shows no independent accuracy advantage yet; the clearest gains are latency and cost.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Reviews 28 papers posted between 19 and 24 September 2026, days after Jev's release.
- Relates them to label-probability classification, constrained decoding, reranking, calibration, and cascades.
- Finds the typed readout has not shown an independent accuracy advantage over comparable label-probability readouts.
- Jev's clearest gains are in latency and cost, while accuracy gaps remain on harder tasks.
- Derives a 14-item evaluation checklist from recurring weaknesses in the reviewed studies.
Watch out
arXiv preprint posted September 26, 2026 (cs.CL, cs.AI, cs.LG; 25 pages) by Lijuan Tang and Yuemeng Zheng. The corpus covers only the first nine days after the release of one hosted model, and the authors say the review should be read as an early evidence map rather than a settled assessment of the model class.
Questions about this entry
Short answers drawn from the linked project or page.
28 arXiv papers posted between 19 and 24 September 2026, shortly after TypeSafe released the commercial typed decision model Jev on 15 September 2026.
It connects them to earlier work on label-probability classification, constrained decoding, reranking, calibration, and model cascades.
In the early literature it has not shown an independent accuracy advantage over comparable label-probability readouts.
In latency and cost, while accuracy gaps remain on harder tasks.
A 14-item evaluation checklist for future typed-decision-model work, derived from weaknesses that recur across the reviewed studies.
Because the evidence covers only the first nine days after the release of one hosted model, so it is not a settled assessment of the model class.
Teams and researchers planning typed-decision-model evaluations, since it is derived from recurring weaknesses in study design found in the first wave of work.
More Jev resources like this
An evaluation of Jev as a semantic decision component in scientific workflows using a harness that assigns arithmetic to code. Across twelve model configurations, twenty source-grounded Choices, and ten scientific cases, Jev matched five configurations at complete semantic correctness.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
David-DL-Space/sys1-evalJavaScript
David-DL-Space/sys1-evalJavaScriptAn evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An empirical study of whether Jev's decision-level savings transfer to task-level gains, across five agents in ALFWorld and AndroidWorld. Across 4,500 task executions, decision cost and latency fall in all five agents, but success declines in two mobile agents and per-outcome cost rises in one.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonA bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.