CommunityJev Benchmarks & Evaluations
Procedural Typed Decisions
572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.
- Format
- Docs
- Published by
- Community
- Author
- tasksource
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- Labels are noise-free by construction: each state spells out the rules, and answers are computed rather than annotated.
- 13 task families from arithmetic and table lookup to policy uncertainty and evidence sufficiency, with 4–60 options.
- Difficulty was calibrated on 40 fresh states per level against Jev 1.13, with kappa falling from 1.00 to 0.36 on the needle task.
- Rebuilding is byte-deterministic: row i of a split is seeded by task:split:i.
Watch out
Synthetic rule-following only, with no human judgments, and difficulty is tied to one Jev version. Jev's probability MAE grows from 0.16 at level 0 to 0.29 at level 4 on the incident_real task, and two families are hard from level 0.
Questions about this entry
Short answers drawn from the linked project or page.
No. The card says answers are computed exactly from rules the state itself spells out, so a model's error is a real error rather than disagreement with a teacher. Jev is used only to set difficulty levels per task.
The generator deliberately varies option counts from 4 to 60 to balance the binary and 4–6-option skew of other Jev datasets, which makes it useful for testing wide-choice behaviour without pre-filtering.
Yes. Rebuilding is byte-deterministic: row i of a split is seeded by task:split:i, so the same generator reproduces the same rows every time.
On 40 fresh states per level against Jev 1.13, with agreement (kappa) falling from 1.00 to 0.36 on the needle task. Difficulty is therefore tied to that one Jev version.
Its probability MAE grows from 0.16 at level 0 to 0.29 at level 4 on the incident_real task, and two of the families are hard from level 0.
No. It is synthetic rule-following only, with no human judgments. Each state spells out the rules and answers are computed exactly, so a model's error is a real error rather than disagreement with a teacher.
13 task families, from arithmetic and table lookup to policy uncertainty and evidence sufficiency, with option counts ranging from 4 to 60.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPython4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript3GitHub stars
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonThe 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
A native Brazilian Portuguese typed-decisions benchmark: seven choice and noul tasks over legislative, legal, scientific, and fact-checking text, with a tuned 4B model at 72.9% mean.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.