Synthetic data is not free labels. It is a teacher's opinion at scale, and the useful questions are what shape each record takes, how the target is represented, and what the student is allowed to learn from it. A corpus that gets those right can train a small decision model; one that gets them wrong teaches the student to imitate a confident argmax.
The public work in this area is unusually transparent about provenance, which makes it a good place to learn the format.
The shape to copy: state plus typed questions
A decision record is a state, one or more typed questions over that state, and a target distribution over the options. The Claude-labelled distillation set is a clear example: 37,932 training states carrying 150,827 questions, plus 1,645 validation states and 7,349 questions. The questions are typed: 38,871 Noul, 65,950 Choice with 2–40 option ids, and 46,006 Score with 3–7 ordered ids.
Workflow states in that set carry eight questions, one of them intentionally unanswerable with label -1. Including a rejection case gives the student somewhere to put uncertainty.
The synthetic typed decisions corpus follows the same shape at a smaller scale: 8,916 items and 28,661 decisions across 314 workflows, split into 6,666 train items over 235 workflows, 1,176 validation over 41, and 1,074 test over 38. The decision mix is 15,585 Choice, 7,762 Noul, and 5,314 Score.
Soft targets beat argmax labels
The most important property of a good synthetic set is that it preserves disagreement. In the synthetic typed decisions corpus, 44% of labels are below 0.7 confidence and only 11% are at least 0.95. The card advises training on the full distributions rather than the argmax. That is the difference between teaching a model "this is the answer" and teaching it "this is how the evidence splits."
The labels themselves are model-derived. Jev 1.13 labelled every kept question, a separate blind DeepSeek pass answered each with probabilities and flagged ill-posed questions, and the answers field averages the two. On 75 blind-labelled questions, that average was closer to the annotator than either model alone, at 0.135 total variation. This is consensus labels with models as the annotators, not human ground truth, and the card says so.
The Claude-labelled set takes a different route to the same end. Its targets are teacher distributions, sharpened label mixtures, or Opus verbalised probabilities, depending on the vote field. The 27B Clef model is the main teacher, with clef-flash as KL anchor and Opus re-judging disputes. Both approaches produce a distribution rather than a single label, which is what a decision model needs.
Three public corpora with different provenance
The Open-Jev typed decision datasets are the most explicit about being a controlled benchmark rather than training truth. The project publishes twelve frozen configs. The default release-v2-redistributable holds 113,568 rows, browser-drone-expansion-v1-redistributable holds 160,797, mailroom-control-v1 holds 114,800, and small controls like citation-control-v1 run 4,000. Every config splits into train, calibration, validation, test, and OOD.
Two cautions from the card. First, release-v2 is contained within the browser-drone expansion, and the overlap paragraph is truncated, so do not sum config totals as if they were disjoint. Second, these are controlled, mostly synthetic tasks with reference labels, not official TypeSafe or Jev training data, and the project is not affiliated with TypeSafe. The repository is CC0 1.0.
The Claude-labelled set is Apache-2.0 and reports that its fine-tune lifts the Decision Index public suite from 52.55 to 54.96 on the same 123,049 rows, a paired McNemar gain of +2,227 net rows at p about 1e-138. It also reports a proxy index of 59.33 versus 57.98 for TypeSafe Jev 1.13 across the same 32 benchmarks. Those are the author's own protocol, not a neutral benchmark, and the states are synthetic. Read them as a reproducibility claim, not an independent ranking.
Distilling into a small open model
JevAlt is a good example of the deployment target. It ships three 4B models (Deem-4B for English, Karar-4B for Turkish, and Wähler-4B for German), LoRA-trained from Intern-Decision-4B. The server implements the Jev API at POST /v1/systemone, so existing clients work by changing the endpoint. Q4_K_M GGUF builds run on a CPU in about 3 GB of RAM.
The reported held-out accuracy is 94.7% English, 96.8% Turkish, and 92.0% German, each compared with Kev-4B. A live test answered 122 of 130 English requests correctly across five cases. The README says the accuracy is measured on the project's own data pipeline, which favours JevAlt, and that calibration should be refit on your own data. The models still follow some hidden instructions planted in input text, so test that behavior before triggering actions.
The oss-jev-distill pack takes a narrower approach: 10,044 rows total, with 7,909 train, 1,074 eval, and 1,061 calibration, plus 1,708 labelled fixtures. The teacher is jev-1.13.0, and the schema is oss-jev-distill-v1 for the one-token Qwen decision track. The card says explicitly not to train on the eval or calibration splits. It does not document the row schema, question types, or label construction, so treat it as a starting pack rather than a specification.
The oss-jev-train-src package is the companion job source: a PEP 723 entry point that declares its own dependencies and downloads the distill pack when /data is not mounted. The four-line card documents packaging, not hardware, hyperparameters, artifacts, or evaluation. The repository files are the authority.
What the evidence says about distillation gains
The search and multi-teacher distillation paper is the most concrete evidence that this works. It distills pairwise Jev judgments into a compact evaluator that runs at every position in Stockfish search, because live calls are too slow. The resulting bot climbs above a 2200 Lichess bullet rating against other bots.
The more useful result is about teachers. Averaging Jev and Qwen3-32B labels beats spending the whole labeling budget on Qwen alone by 9.6 Elo, with a 95% interval of 4.3–14.9. In passage reranking, Jev's labels alone train a reranker that matches Qwen's, from 21 minutes of API calls versus 5.1 GPU-hours. A second answer from the same judge is no substitute for a second teacher, and Jev was Qwen's strongest partner among the models tested.
Those results are self-reported for one chess bot and one reranking setup, and the abstract gives no full Elo spread. The early evidence audit supplies the wider caution: across 28 papers posted in the first nine days after Jev's release, the typed readout had not shown an independent accuracy advantage over comparable label-probability readouts. The clearest gains were latency and cost, while accuracy gaps remained on harder tasks. The audit derives a 14-item evaluation checklist and calls itself an early evidence map, not a settled assessment.
Split by workflow and hold out honestly
The synthetic typed decisions corpus splits by workflow, so validation and test use decision applications never seen in training. That is the right default. A random split over items from the same workflow measures memorization of the workflow, not transfer to a new one.
Three more checks before training. Look for overlap between the benchmark and the training data; the JEV Decision Index results card reports a training-overlap check with no whole-suite state, instruction, or option in the training data. Do not sum configs that overlap. And check the license at the level you need: repository-level CC0 or Apache-2.0 does not necessarily settle the terms of every source component.
The jaggedness list is the final filter. A synthetic set inherits the teacher's quirks, including unreliable counting and literal reading, and averaging two models does not remove a shared bias. Label a small sample yourself and compare. The fine-tuning and calibration article covers what to do after training, and distillation recipes covers the teacher-student loop. The corpus is only the beginning of the evidence.
