Docs
CommunityJev Benchmarks & Evaluations
Synthetic Typed Decisions
8,916 LLM-written items with several typed questions each, following the Jev request shape: 28,661 choice, noul and score decisions across 314 workflows, labelled by averaged model probabilities.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- tasksource
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- 8,916 items and 28,661 decisions split by workflow: 6,666/21,190 train, 1,176/3,996 validation and 1,074/3,475 test across 314 workflows.
- Decisions are 15,585 choice, 7,762 noul and 5,314 score; labels are below 0.7 confident on 44% of them and at least 0.95 on 11%.
- Answers average Jev 1.13's probabilities with a blind check's; on 75 blind-labelled questions the average was closer to the annotator at 0.135 total variation.
- Teacher distributions include upstage/solar-decide across up to 26 options and respan/span-01 for yes/no questions.
- Splits are by workflow, so validation and test use decision applications never seen in training; Apache-2.0.
Watch out
Card reviewed 2026-10-17; items are synthetic and the labels are model-derived — Jev 1.13 and a blind check, averaged — so they are not human ground truth, a few questions lean on unstated standards, and workflows differ by design between train and test. The dataset is independent of TypeSafe and OpenJev.
Questions about this entry
Short answers drawn from the linked project or page.
LLM-written items such as emails, chats, forum threads and case notes, each with several typed questions over the same item, following the Jev request shape of choice, noul and score decisions on judgment that rules cannot compute.
Jev 1.13 labelled every kept question and a separate blind DeepSeek pass answered each with probabilities and flagged ill-posed questions; the answers field averages the two, and questions where they confidently disagreed were dropped.
Splits are by workflow: 6,666 train items over 235 workflows, 1,176 validation items over 41 and 1,074 test items over 38, so validation and test use decision applications never seen during training.
Mostly uncertain by design: labels are at least 0.95 confident on 11% of decisions and below 0.7 on 44%, and the card advises training on the full distributions rather than the argmax.
They are companions from the same project: anything formalizable by rules, thresholds, counting or lookups is left to procedural-typed-decisions, whose labels are exact, while this dataset covers meaning, intent, tone, stance and routing judgment.
Apache-2.0, and the card states the dataset is independent — not an official TypeSafe Jev dataset and not produced by or affiliated with TypeSafe or OpenJev.
The items are synthetic and teacher models wrote them; a few questions still rely on an unstated standard such as standard safety procedures, so labels should be treated as model opinions averaged for consensus rather than ground truth.
More Jev resources like this
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Twelve frozen typed-decision configs from the independent Open-Jev project, publishing controlled and mostly synthetic tasks with probability or ordinal targets: from the 113,568-row default release to the 160,797-row browser and drone expansion. CC0 1.0; English, Chinese and Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.