CommunityJev Benchmarks & Evaluations
S1MB Leaderboard
The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.
- Format
- Playground
- Published by
- Community
- Author
- hotchpotch
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
- Platforms
- Browser
- Capabilities
- Open source
Highlights
- Choice, Noul, and Score results across more than 100 tasks for roughly 100 model repositories.
- Missing or failed measurements stay visible as gaps instead of being estimated.
- Scores live in the public hotchpotch/s1mb-result dataset; the space embeds compact display data rebuilt on deploy.
- The board spans Laya, Kev, Open-Jev, Darwin, Bekko, Clef, GLiNER, rerankers, and general encoders.
Watch out
The card says scores do not establish unseen-task generalization or training-data non-overlap, and numbers only change when the display data is regenerated and the space redeployed. It is a viewer, not an inference runner.
Questions about this entry
Short answers drawn from the linked project or page.
No. It is a viewer over saved measurements on a CPU runtime; nothing loads weights or runs inference. Results come from the public hotchpotch/s1mb-result dataset, built from hotchpotch/s1mb-dataset.
Because missing or failed measurements are preserved as visible gaps rather than imputed, which the card frames as part of the board's honesty. Updating scores means regenerating the display data and redeploying the space.
From the public hotchpotch/s1mb-result dataset, built from hotchpotch/s1mb-dataset. The space embeds compact display data that is rebuilt on deploy, and results are regenerated rather than estimated.
Around 100 model repositories across more than 100 specialized tasks, spanning Laya, Kev, Open-Jev, Darwin, Bekko, Clef, GLiNER, rerankers and general encoders.
No. The card says scores do not establish unseen-task generalization or training-data non-overlap, so treat them as task-specific measurements rather than a generalization guarantee.
Numbers only change when the display data is regenerated and the space redeployed, because the board reads saved measurements rather than running inference live.
Choice, Noul and Score, across the more than 100 tasks, with results attributed per model repository.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
2GitHub stars
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPython2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
A native Brazilian Portuguese typed-decisions benchmark: seven choice and noul tasks over legislative, legal, scientific, and fact-checking text, with a tuned 4B model at 72.9% mean.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
Interface, size, license, claims — in that order. A selection guide across the system-one catalog, from a 706K form scorer to 27B Jev students.
Jev Practices & Patterns#open-models#evaluation#benchmarks
Read article
Hundreds of repositories carry the system-one tag, but the ecosystem is a dozen ideas in different sizes. A tour of the families, the licenses, and what a download does not include.
Jev Models & Reimplementations#open-models#hugging-face#local
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.