CommunityJev Benchmarks & Evaluations
Jev-Bench
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
- Format
- Docs
- Published by
- Community
- Author
- Praveenrajus
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 22 configs and 166,054 rows, of which 22,773 test records were scored across 46 models in v0.1.1.
- Four configs carry human label distributions: go_emotions, measuring_hate_speech, civil_comments and chaosnli.
- TypeSafe Jev 1.13.0 baseline: 0.733 accuracy and ECE 0.113 on the shared test records.
- Best open entry, Qwen-AgentWorld-35B-A3B with a fine-tuned decision readout, reaches 0.770 accuracy and ECE 0.055 in one seed.
- Rows are uniform samples with seed 20260920 so natural base rates are preserved; test caps at 1,000 rows per config.
Watch out
The dataset-level licence is 'other' with per-config terms (cc-by-4.0, cc-by-3.0, mit, cc-by-sa-4.0, plus research-use sources like MNLI and Yelp). Human distributions exist only on the four vote-share configs, the best fine-tuned entries are one-seed results, and the card describes Jev 1.13.0's own baseline as over-confident where people disagree.
Questions about this entry
Short answers drawn from the linked project or page.
Three: choice over K options, score over K ordered levels with an expected score, and noul as a single P(yes). Every row is a state, question and label triple in exactly the wire format a System One model consumes.
22 configs and 166,054 rows, including 22,773 test records scored across 46 models. Test caps at 1,000 rows per config except civil_comments at 2,000, and ChaosNLI is included in full.
Four carry the annotator vote shares behind each label: go_emotions, measuring_hate_speech, civil_comments and chaosnli, where ChaosNLI items were re-annotated by 100 crowdworkers each.
Jev 1.13.0 scores 0.733 accuracy with ECE 0.113. The best open result, Qwen-AgentWorld-35B-A3B with a fine-tuned readout and coherence penalty, reaches 0.770 and ECE 0.055, and at 9B Qwen3.5-9B reaches 0.763 and 0.056.
The card lists licence 'other' because terms are per config: cc-by-4.0, cc-by-3.0, mit and cc-by-sa-4.0, plus research-use terms for MNLI and Yelp and unspecified terms for the Stanford Sentiment Treebank.
Rows are uniform random samples with seed 20260920 so the benchmark does not shift base rates; test is at most 1,000 per config, validation up to 500 and train up to 8,000 so tuning stays in distribution.
The card reports Jev ahead on the six held-out sources at 0.835 against 0.828 for the best open model and 0.840 for Perplexity's hosted pplx-decider-v1-27b, and on applying a rule stated in the question at 0.924 against 0.80.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript17GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.