Docs
CommunityJev Benchmarks & Evaluations
Calibrated Typed Decisions
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- m-lagnajit
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 16,995 train, 1,799 validation, 1,797 test and 1,000 held-out rows over choice and noul questions on short English texts.
- The held-out split is 1,000 expert-labelled PubMedQA yes/no/maybe questions from a source and decision kind seen nowhere else.
- v2.1 adds corrected_soft_label, which swaps the teacher's top pick with the right answer where they disagree, and a flag column.
- Teacher top-pick error ranges from 3.3% (amazon_polarity noul) to 26.7% (scicite noul); GoEmotions labels use human rater votes.
- Calibration cuts ECE on test: scicite choice 0.190 to 0.030, clinc_oos noul 0.211 to 0.038; released under CC BY 4.0.
Watch out
Card reviewed 2026-10-17; soft labels are teacher- or rater-derived rather than empirical uncertainty, the card warns not to train on or fit dials against the held-out split, and the ECE and accuracy figures are the dataset authors' own measurements on the test split. Upstream sources keep their own licences.
Questions about this entry
Short answers drawn from the linked project or page.
A dataset of typed decisions — multiple choice and yes/no — over short English texts where every row has a human gold label and a calibrated soft label, plus the raw teacher log-probabilities and dials to rebuild the soft label your own way.
Two columns: corrected_soft_label, which keeps the teacher's numbers but gives the largest value to the right answer where the teacher's top pick was wrong, and soft_label_corrected, true exactly on the rows the swap changed.
From a calibrated teacher distribution (teacher_probs, fit per source and question type on validation) except on GoEmotions, where soft labels come from raw human rater votes because the teacher's top pick disagreed with the human answer 38.5% of the time.
As a test only: 1,000 expert-labelled PubMedQA questions with yes, no or maybe answers from a label set and domain unseen in training. The card says do not train on it and do not fit dials on it.
On test, calibrated ECE falls sharply against the raw teacher: 0.190 to 0.030 on scicite choice, 0.211 to 0.038 on clinc_oos noul and 0.398 to 0.088 on GoEmotions choice, while accuracy on choice items mostly stays the same.
CC BY 4.0 for the dataset, while the sources carry their own terms: CLINC150 CC BY 3.0, GoEmotions Apache-2.0, BANKING77 and MASSIVE CC BY 4.0, Amazon polarity and SciCite Apache-2.0 per their repositories, and PubMedQA held out for testing.
Rows carry instructions, state, options with descriptions, a gold answer and probability targets across options, matching the typed-decision request shape; the noul rows encode yes/no with the false/true option pair.
More Jev resources like this
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
The training and validation data behind jaeyong2's clef-flash-jev-claude-distill fine-tune: 37,932 states and 150,827 typed questions with soft target distributions, plus a 19,025-record language pool. The card reports the fine-tune lifts the Decision Index public suite from 52.55 to 54.96. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
8,916 LLM-written items with several typed questions each, following the Jev request shape: 28,661 choice, noul and score decisions across 314 workflows, labelled by averaged model probabilities.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.