User repository on GitHub
System One Mosaic Benchmark
- Forks
- 6
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
S1MB
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
- Stars
- 2
- Language
- Python
- License
- MIT
- Published by
- Community
- Last verified
- Oct 2026
About this project
S1MB Leaderboard · Explainer article · Quick start · Results dataset · Evaluation dataset · Documentation · Submit results. S1MB provides a practical reference point for comparing System One Decision Models on the same inputs and metrics. These models use text and instructions to select an option (Choice), make a yes/no judgment (Noul), or assign a numeric score using supplied criteria (Score). The leaderboard lets you compare overall results and inspect which tasks each model handles well or struggles with.
Quoted from the project's README. Read the full README.
Works forJev for Evaluation
Used inJev for Software & DevTools
- Languages
- PythonTypeScript
- Platforms
- Browser
- Capabilities
- Open source
Highlights
- English suite: 137 benchmarks across 106 dataset subsets, including six synthetic benchmarks probing varied instructions, contexts and decision criteria.
- Metrics: target mass for Choice, Brier score for Noul, normalized expected-value MAE for Score, combined into a baseline-adjusted Task Avg on a 0-100 scale.
- Borda Score is the viewer's default ranking across complete models and can change when the model roster changes.
- Use supported adapters including TypeSafe/Jev and Bekko, or implement the typed adapter contract for another model.
- Results retain exact dataset and model revisions, evaluator provenance and inference settings; missing measurements stay visibly incomplete.
Watch out
MIT licensed for project code; the README states scores do not establish unseen-task generalization or training-data non-overlap, and adding models depends on volunteer time. Datasets, weights and upstream packages keep their own licenses.
Questions about this entry
Short answers drawn from the linked project or page.
It runs System One decision models -- models that pick an option (Choice), answer yes/no (Noul) or assign a score from authored criteria (Score) -- on the same 137-benchmark English suite, and publishes per-task and aggregate results on a Hugging Face leaderboard.
Yes. The Python 3.11 evaluator supports a typed adapter contract; you validate measurements locally and submit them as a results-dataset PR. Requests to evaluate specific models depend on volunteer time and resources, as noted in the request policy.
Public checks use synthetic fixtures and need no tokens, models or GPU; a full run uses the Python 3.11 evaluator with uv, and datasets download on demand unless you pass --offline-dataset.
Choice uses target mass at the selected alternative, Noul uses a Brier score, and Score uses normalized expected-value MAE; these combine into a baseline-adjusted Task Avg on a 0-100 scale.
It is the viewer's default ranking across complete models with equal weight per benchmark, and unlike Task Avg it can change when the model roster changes; both need complete coverage in the selected scope.
It ships adapters for TypeSafe/Jev and Bekko, and you can implement the typed adapter contract for another model, then validate measurements and submit them as a results-dataset pull request.
No. The README states the scores do not establish unseen-task generalization or training-data non-overlap, and missing or failed measurements stay visibly incomplete rather than being filled in.
More Jev resources like this
24kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython7GitHub stars
Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter Notebook3GitHub stars
A benchmark that steers decision models through 100 key-and-door grid puzzles with one Jev Choice per move, scored against an exact BFS solver; it found Jev picks an optimal move about 90% of the time yet finishes only 57 of 100 games.
Jev Benchmarks & Evaluations#community#benchmark#gridworld
2GitHub stars
Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.
Jev Benchmarks & Evaluations#community#benchmark#calibration
Community
jjd-lab/jev-synthetic-surveyPython
jjd-lab/jev-synthetic-surveyPythonFrom the Jev community
Posts from builders shipping with Jev right now.