CommunityJev Benchmarks & Evaluations
Typed Decisions Benchmark
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
- Format
- Docs
- Published by
- Community
- Author
- LocalLLaMA
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
- Capabilities
- Open source
Highlights
- Every row's state and questions are exactly the body of a POST /v1/systemone request; a case id or state cannot appear in both splits.
- Gold is the mean of three teacher samples at temperature 0.7, which is why the self-agreement ceiling is 0.735 rather than 1.0.
- Leaderboard top as published: meraGPT Decider 1 at 0.768, Liquid AI d1 at 0.742, Jev 1.13.0 at 0.727.
- Jev's KL is 1.442 despite 0.727 accuracy — the card's own example of a very sharp distribution scoring well on accuracy.
Watch out
Independent of TypeSafe and not a reproduction of Jev. Gold agreement is not correctness, so a better model can score lower wherever the teacher is wrong, and per-question ceilings range from 0.560 to 0.937.
Questions about this entry
Short answers drawn from the linked project or page.
No. The card says it follows the System One primitives but is not affiliated with TypeSafe and does not reproduce the Jev model. The gain is that the request shape replays against any compatible API, with 2,000 decisions across four workflows.
Because gold is the mean of three teacher samples: 0.735 is the teacher's self-agreement, so scores above it mean learning teacher quirks rather than the task. The card says specialists and general models are not comparable on the same number.
Four: agent_trace_observability, customer_service, invoice_processing, and security_incidents, each a separate 400-row subset, for 2,000 decisions total in rows shaped as raw /v1/systemone request bodies.
A case id or state cannot appear in both splits, so the same state never shows up in train and test; each subset holds 300 train rows and 100 test rows.
As published, meraGPT Decider 1 leads at 0.768, then Liquid AI d1 at 0.742 and Jev 1.13.0 at 0.727. Gold is teacher agreement, not correctness, so a lower number is not automatically worse.
Yes. Each row's state and questions are exactly the body of a POST /v1/systemone request, so it replays against any compatible decision API, and case ids and states are kept out of both splits.
It varies: per-question teacher ceilings range from 0.560 to 0.937, so the headline 0.735 agreement ceiling is an average rather than a hard limit on every item.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
A native Brazilian Portuguese typed-decisions benchmark: seven choice and noul tasks over legislative, legal, scientific, and fact-checking text, with a tuned 4B model at 72.9% mean.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPythonFrom the Jev guides
Original write-ups on what this entry is for.
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
Hundreds of repositories carry the system-one tag, but the ecosystem is a dozen ideas in different sizes. A tour of the families, the licenses, and what a download does not include.
Jev Models & Reimplementations#open-models#hugging-face#local
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.