CommunityJev Benchmarks & Evaluations
Jev Distill Corpus v3
The 741,000-row distillation corpus behind AutoTrust's Jev students: choice, score, and noul rows with full target distributions across yuri and openjev source streams, Apache-2.0.
- Format
- Docs
- Published by
- Community
- Author
- SargeDev
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
- Capabilities
- Open source
Highlights
- 741,000 rows split into train, validation, calibration, test, ood, and a 30,000-row test_set_30k.
- Each row carries id, kind, options, target, state, question, domain, family, and source — targets are full distributions, not labels.
- Kinds span the System One primitives with 2–16 options, across 66 domains and 14 families from fact-checking to multi-agent coordination.
- Three source streams — yuri_v1, yuri_v3, and openjev_v2 — so provenance is traceable per row.
Watch out
The calibration split is a separate split for a reason: temperature fitting belongs on it, not on the test or ood rows. The corpus is training data for AutoTrust's students, not a benchmark, and some source rows repeat distributions at 0.5/0.5 where the teacher was uncertain.
Questions about this entry
Short answers drawn from the linked project or page.
It is the 13,800-row split intended for fitting temperatures and biases without touching test or ood data, matching what the AutoTrust cards describe when they say head bias and temperature are applied client-side after training.
It is the corpus linked from the JEV-9B and JEV-27B model cards, licensed Apache-2.0 with 741,000 rows. The students themselves note that the training pipeline is not shipped, so the corpus reproduces the data shape rather than the full recipe.
Apache-2.0. The corpus is public training data linked from the AutoTrust student model cards, with the openjev_v2 source stream additionally CC0.
id, kind, options, target, state, question, domain, family, and source. Targets are full distributions rather than labels, and kinds span the System One primitives with 2-16 options.
741,000 rows in total, split into train (656k), validation (14.1k), calibration (13.8k), test (14.3k), ood (13.1k), and a 30,000-row test_set_30k used for held-out evaluation.
yuri_v1, yuri_v3, and openjev_v2, so provenance is traceable per row. yuri_v3 carries Jev's output distributions, openjev_v2 programmatic ground-truth labels, and yuri_v1 placeholder labels.
66 domains and 14 families, from fact-checking to multi-agent coordination, including knowledge, chemistry, process safety, and spatial reasoning tasks across the System One primitives.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
24kGitHub stars
An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.
Jev Benchmarks & Evaluations#community#typescript#agents
Community
different-ai/openworkTypeScript
different-ai/openworkTypeScript4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython2GitHub stars
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPython3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonFrom the Jev guides
Original write-ups on what this entry is for.
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
Hundreds of repositories carry the system-one tag, but the ecosystem is a dozen ideas in different sizes. A tour of the families, the licenses, and what a download does not include.
Jev Models & Reimplementations#open-models#hugging-face#local
Read article
From the Jev community
Posts from builders shipping with Jev right now.