CommunityJev Benchmarks & Evaluations
Jev Decision Index
An unofficial Hugging Face space that tracks open Jev reproductions and scores the ones that finished a frozen benchmark suite.
- Format
- Playground
- Published by
- Community
- Author
- multimodalart
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
Highlights
- The news tab groups work into decoding, diffusion, trained weights, prior art, and explainers.
- Decision Index 0.2 averages 40 benchmarks in five equal areas. An entrant needs all 40 to be listed.
- The headline score is chance-corrected: 0 is random, 100 is perfect. Raw accuracy stays on the model page.
- The README says TypeSafe's weights, the RLCD algorithm, and a calibration match are still not open.
Watch out
Community-run and not affiliated with TypeSafe. Engagement numbers on the news tab are a 24 September 2026 snapshot.
Questions about this entry
Short answers drawn from the linked project or page.
It is community-run and unaffiliated with TypeSafe, per the caveats, and the description calls it unofficial. The captured source page shows only the space title and an engagement count, so it does not document the methodology behind the scores.
The suite is gated. The highlights say Index 0.2 averages 40 benchmarks across five equal areas and that an entrant needs all 40 to be listed, which is what keeps a cross-model comparison from being apples to oranges.
No. The highlights say the headline number is chance-corrected, with 0 as random and 100 as perfect, and that raw accuracy stays on the individual model page. Check the raw figure before quoting the index value.
Not the same model. The highlights say TypeSafe's weights, the RLCD algorithm and a calibration match are still not open, so what the index scores is a set of reimplementations rather than the original checkpoint.
The news tab groups work into five buckets: decoding, diffusion, trained weights, prior art and explainers. That is how the space separates different kinds of open Jev reproduction when it lists them.
A snapshot. The caveat states the engagement numbers on the news tab were captured on 24 September 2026, so like and view counts will be stale rather than live.
It must have completed the whole suite: Decision Index 0.2 averages 40 benchmarks across five equal areas, and an entrant needs all 40 to be listed. That gating is what keeps a partial run out of the ranking.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsLinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsHacker News: A one-token logprob wrapper that also classifies webcam frames.
allanrbo151 pointsA single function Jev-like wrapper for LLMs, including vision modelsRead the thread on Hacker NewsRelated terms
Glossary definitions related to this entry.
More Jev resources like this
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
24kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript298GitHub stars
A self-hosted implementation of TypeSafe's Jev System One API powered by the 400M-parameter GLiFormer encoder: it serves choice, score, and noul and drops into the official typesafe-sdk via TYPESAFE_BASE_URL, but trails Jev on reasoning-heavy tasks.
Jev Practices & Patterns#community#python#open-models
Community
logan-markewich/jeffPython
logan-markewich/jeffPythonFrom the Jev guides
Original write-ups on what this entry is for.
Interface, size, license, claims — in that order. A selection guide across the system-one catalog, from a 706K form scorer to 27B Jev students.
Jev Practices & Patterns#open-models#evaluation#benchmarks
Read article
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
Hundreds of repositories carry the system-one tag, but the ecosystem is a dozen ideas in different sizes. A tour of the families, the licenses, and what a download does not include.
Jev Models & Reimplementations#open-models#hugging-face#local
Read article
From the Jev community
Posts from builders shipping with Jev right now.