CommunityJev Benchmarks & Evaluations
Tasksource Jev Typed-Decisions
2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
- Format
- Docs
- Published by
- Community
- Author
- tasksource
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- 2.5 million decisions from 667 sources, with a 1.06M-row default config plus filtered-full, full, and a vision variant.
- Supervision is real source labels — the card says Jev and Liquid D1 only score sources for mix weighting and audits.
- Every row carries license and license_use fields, so commercial filtering is a data operation rather than a guess.
- Soft targets are kept for sources with at least five annotators, including STS, ChaosNLI, and civil_comments.
Watch out
License is listed as 'other' because per-row terms still apply, and license_use is a best-effort aid, not legal advice. The vision config is a mixed-license research pilot: of 66,085 training rows counted 2026-10-09, 26,987 were commercial, 13,000 non-commercial, and 26,098 unresolved.
Questions about this entry
Short answers drawn from the linked project or page.
No. The card says labels come from established datasets and that Jev and Liquid D1 are used only to weight and audit sources. The recast is independent of TypeSafe and OpenJev, and the vision rows are a separate research pilot.
Only where licenses allow: filter on license_use == 'commercial'. The card warns the field is a best-effort aid rather than legal advice, and the vision config is explicitly not commercial-cleared.
A 1.06M-row default config alongside filtered-full, full, and a vision variant, drawn from 667 sources. Every row carries license and license_use fields, so commercial filtering is a data operation rather than a guess.
From established datasets' real source labels. The card says Jev and Liquid D1 are used only to score sources for mix weighting and audits, so the supervision is not model-generated.
For sources with at least five annotators, including STS, ChaosNLI, and civil_comments, the soft annotator targets are kept rather than collapsed to a single hard label.
Not all of it. The card counted 66,085 training rows on 2026-10-09: 26,987 commercial, 13,000 non-commercial, and 26,098 unresolved, and calls the vision config a mixed-license research pilot.
license_use is a best-effort aid, not legal advice, and the dataset is listed as 'other' because per-row terms still apply. Filter on license_use == 'commercial' and verify each source yourself.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPython4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript3GitHub stars
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonThe 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
A native Brazilian Portuguese typed-decisions benchmark: seven choice and noul tasks over legislative, legal, scientific, and fact-checking text, with a tuned 4B model at 72.9% mean.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
Hundreds of repositories carry the system-one tag, but the ecosystem is a dozen ideas in different sizes. A tour of the families, the licenses, and what a download does not include.
Jev Models & Reimplementations#open-models#hugging-face#local
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.