Docs
CommunityJev Benchmarks & Evaluations
Jev Decision Distillation Data (Claude-Labelled)
The training and validation data behind jaeyong2's clef-flash-jev-claude-distill fine-tune: 37,932 states and 150,827 typed questions with soft target distributions, plus a 19,025-record language pool. The card reports the fine-tune lifts the Decision Index public suite from 52.55 to 54.96. Apache-2.0.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- jaeyong2
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- 37,932 train states with 150,827 questions; validation holds 1,645 states and 7,349 questions.
- The fine-tune reportedly lifts Decision Index public from 52.55 to 54.96 on the same 123,049 rows (McNemar +2,227).
- Card-reported proxy index 59.33 versus 57.98 for TypeSafe Jev 1.13 across the same 32 benchmarks.
- Question types: noul (38,871), choice with 2-40 ids (65,950) and score with 3-7 ordered ids (46,006).
- Teachers: the 27B Clef model as main, clef-flash as KL anchor, plus Qwen3.5-27B and Opus re-judges.
Watch out
Card reviewed 2026-10-17; the fetched card text truncates inside the language-judgement balancing section, so the rest of that subsection is not covered, and the Decision Index and proxy-index figures compare the fine-tune against Jev 1.13 on the author's own protocol rather than a neutral benchmark. States are synthetic, and 16,809 train questions carry label -1.
Questions about this entry
Short answers drawn from the linked project or page.
The training and validation data behind jaeyong2/clef-flash-jev-claude-distill, a fine-tune of Cloudflare/clef-flash (9B). Each record is a decision state with one or more typed questions and a soft target distribution over the options.
On the Decision Index public suite it raises the score from 52.55 to 54.96 on the same 123,049 rows, with paired McNemar +2,227 net rows at p about 1e-138, and on the same 32 benchmarks it reports a proxy index of 59.33 versus 57.98 for TypeSafe Jev 1.13.
train.jsonl with 37,932 states and 150,827 questions, validation.jsonl with 1,645 states and 7,349 questions from the same generators, and language_pool.jsonl with 19,025 language-judgement records including teacher distributions and the Opus re-judge result.
noul (exactly two language-fixed yes/no options, where the first probability is the yes side), choice (2-40 unordered option ids) and score (3-7 ordered ids, low to high). Workflow states carry eight questions, one of them unanswerable with label -1.
The 27B Clef model is the main teacher, with clef-flash as the KL anchor, plus Qwen3.5-27B on earlier workflow records and claude-opus-5-5 re-judging disputed labels. Targets are teacher distributions, sharpened label mixtures, or Opus verbalised probabilities, per the vote field.
English, Korean, Japanese and Chinese: train states split roughly 11,203 English, 9,005 Korean, 8,889 Japanese and 8,835 Chinese, with the question and option language matching the state. Language-judgement items are English-heavy at 55 percent.
Apache-2.0. The fetched card text truncates inside the language-judgement balancing section, so the remainder of that subsection is not covered here; states are synthetic, and 16,809 train questions carry label -1 with no label push.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Both Jev 1.13 and 1.14 score 29 of 30 on 30 Turkish buyer and seller messages
More Jev resources like this
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Teacher-labelled distillation pack for the one-token Qwen decision track: 10,044 rows across train, eval and calibration JSONL files, plus 1,708 labelled fixtures. Labels come from jev-1.13.0, and the card says not to train on the eval or calib splits. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPythonLangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.