CommunityJev Benchmarks & Evaluations
Open-Jev v1.1 Data
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
- Format
- Docs
- Published by
- Community
- Author
- ZefanCai
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- Published projection counts: 147,139 train, 26,675 calibration, 25,413 validation, 43,125 test and 84,267 OOD records.
- 2,053 Wikispeedia rows are omitted because a separate redistribution licence for the archived graph and path data was not verified.
- Adds 51,200 original controlled decision rows across 80 authored policy families and 6,400 underlying scenarios.
- 101,207 WANLI-derived English NLI records, from 82,045 train rows to 4,627 OOD, remain CC BY 4.0.
- The complete mixture manifest SHA-256 is 7b26f948d2ae11f20d7a18be437d1ada616fc479e1596ae94be7596787fe4e54.
Watch out
Licence is source-dependent: WANLI text and derivatives stay CC BY 4.0, original generated Open-Jev content is CC0 1.0 and code is MIT, while TypeSafe-derived question descriptions have unverified terms. Published metrics are not measurements on this filtered projection, targets are not automatically empirical uncertainty, and the v1.1 name does not imply every model size finished training on it.
Questions about this entry
Short answers drawn from the linked project or page.
2,053 Wikispeedia records are omitted for licensing reasons, 1,500 train, 89 calibration, 69 validation, 176 test and 219 OOD, leaving 326,619 of the complete mixture's 328,672 rows.
The projection holds 147,139 train, 26,675 calibration, 25,413 validation, 43,125 test and 84,267 OOD rows; the 27B stage trained on the complete 148,639-row training split for 37,160 optimizer steps.
51,200 original controlled decision rows across eight authored domains and 80 policy families, plus 101,207 WANLI-derived English NLI records; remaining records replay older Open-Jev controls capped at 1,500 rows per source.
kind is choice, noul or score, and targets are distributions over the supplied options, with noul encoded as [P(no), P(yes)]. The card says metadata.target_basis records whether labels come from rules, known stochastic construction, human annotation or uniform optimal sets.
The model input is state, question, kind and options; dataset metadata, control latents and target distributions are supervision and provenance fields rather than input.
WANLI text and derivatives remain CC BY 4.0, original generated Open-Jev content is dedicated CC0 1.0 and source code is MIT; the card states these terms do not relicense external wording, assets, model weights or datasets.
With separately obtained wikispeedia-v1 files, reproduce/restore_original_mixture.py selects the retained Wiki subset by canonical hashes, restores row order and verifies split checksums, but it does not download the archive or grant rights to it.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPython7GitHub stars
A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.