CommunityJev Benchmarks & Evaluations
Typed Decision Leaderboard
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
- Format
- Playground
- Published by
- Community
- Author
- mayafree
- Last verified
- Oct 2026
- Platforms
- Browser
- Capabilities
- Open source
Highlights
- Every entrant is scored on the same 2,018 items with the same labels, and never re-implemented for scoring.
- Top three sit in a statistical tie by paired bootstrap: JEV 0.7350, ZTC-Judge-27B 0.7289, Darwin-397B-ZTC 0.7272.
- On 1,939 answers from Claude Haiku 4.5, ZTC-Judge-27B v2 scored 0.7752 against JEV's 0.7521, with a confidence interval excluding zero.
- A label-shuffle control reads 0.4996, which the authors present as validating the harness.
Watch out
Test-item texts stay private under source licenses (KMMLU CC BY-ND, CLIcK, GPQA); scores, labels, and grading code are open. API and local divisions are never mixed, and off-axis systems such as game bots and routers are listed outside the ranking.
Questions about this entry
Short answers drawn from the linked project or page.
No. The card says the 2,018 item texts stay private under their source licenses, but the scores, labels, and grading code are published so the harness can be inspected and reproduced up to the data boundary.
By paired bootstrap confidence intervals rather than editor judgment: the ZTC minus JEV interval is -0.034 to +0.020, which is why three systems share the top position on 0.7350, 0.7289, and 0.7272.
Entrants are scored as answer verifiers on the same 2,018 items with the same labels, ranked by weighted AUC, and each is run on its maker's own code rather than a reimplementation.
On 1,939 answers from Claude Haiku 4.5, ZTC-Judge-27B v2 scored 0.7752 against Jev's 0.7521, with a confidence interval that excludes zero, though the two sit in a statistical tie on the full set.
Yes, a label-shuffle control reads 0.4996, which is essentially chance; the authors present it as evidence the harness measures something real rather than leaking labels.
No. The API and local divisions are never mixed, and off-axis systems such as game bots and routers are listed outside the ranking rather than folded into the leaderboard.
KMMLU under CC BY-ND, CLIcK, and GPQA. Those licenses are why the 2,018 item texts stay private while the scores, labels, and grading code are open.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
24kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript2GitHub stars
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPython2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
Interface, size, license, claims — in that order. A selection guide across the system-one catalog, from a 706K form scorer to 27B Jev students.
Jev Practices & Patterns#open-models#evaluation#benchmarks
Read article
Hundreds of repositories carry the system-one tag, but the ecosystem is a dozen ideas in different sizes. A tour of the families, the licenses, and what a download does not include.
Jev Models & Reimplementations#open-models#hugging-face#local
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.