Distillation turns a teacher's decisions into a student's weights. For decision models the interesting part is what the teacher supplies. A hard label says "B." A soft distribution says "B with 0.72, A with 0.19, C with 0.09," and that shape teaches calibration. Open recipes differ on teacher identity, label type, data mixture, and how much of the student is trained.
What the teacher actually supplies
The Claude-labelled distillation set is the most detailed example: 37,932 training states with 150,827 questions, plus 1,645 validation states and 7,349 questions, each carrying a soft target distribution. The main teacher is a 27B Clef model, with clef-flash as a KL anchor, Qwen3.5-27B on earlier records, and Claude Opus re-judging disputed labels. The fine-tune built on it reportedly lifts the Decision Index public suite from 52.55 to 54.96 on the same 123,049 rows and reports a proxy index of 59.33 against 57.98 for Jev 1.13. Those are the author's own protocol, not a neutral benchmark, and 16,809 train questions carry label -1.
oss-jev-distill is the smaller pack: 10,044 rows, split 7,909 train, 1,074 eval, and 1,061 calibration, plus 1,708 labelled fixtures. The teacher is jev-1.13.0 for every record, so it reproduces one Jev version rather than a teacher mixture. The card says not to train on eval or calibration, but is thin on schema: the row format and label derivation are not described, and no student evaluation is published.
oss-jev-train-src is a four-line source package for running the oss-jev job on Hugging Face Jobs. It uses a PEP 723 entry point and downloads the distill pack when /data is not mounted, but documents no recipe, hardware, hyperparameters, outputs, or results. Useful plumbing, not proof.
Multi-teacher and search distillation
From Probabilities to Decisions is the strongest argument for treating teachers as an ensemble. The authors distil pairwise Jev judgments into a compact evaluator that runs at every position in Stockfish search, because live calls are too slow. The bot climbs above a 2200 Lichess bullet rating. In reranking, Jev's labels alone train a reranker that matches Qwen's, from 21 minutes of API calls instead of 5.1 GPU-hours.
The multi-teacher result transfers. Averaging Jev and Qwen3-32B labels beats spending the whole labeling budget on Qwen alone by 9.6 Elo, with a 95% interval of 4.3 to 14.9, and the gain replicates on fresh openings. A second answer from one judge is no substitute for a second teacher. The lesson is diversity: two correlated teachers duplicate their errors, while two differently trained teachers give a useful disagreement signal.
Releasing the Inner Jev points in a related direction. Open instruction-tuned models already read answers from letter logits without training; LoRA plus a decision head then moves Gemma 4 31B from 79.6 to 84.4 on OpenDecisions and to 81.5 on JevBench v1.7. The head starts from a slice of the language-model head, close to the teacher's readout geometry.
Label-free and self-distilled variants
FunDM removes the gold-label requirement. It trains a small head to match a frozen language model's answer probabilities with KL divergence, using 24 of Qwen3-8B's 36 layers and skipping the LM head. The +1T variant reaches 75.0% mean accuracy over eight benchmarks against 75.7% for the full model, with 1.56–1.57 times batched vLLM throughput and about one-third lower peak GPU memory.
WorkMagic's churn model is self-distillation in production clothing. A LoRA on Qwen3-8B trains from the AnyJev teacher Decider(level=L0) with no human labels, with loss on answer tokens in the AnyJev readout format. The card reports no measured results; accuracy, F1, and expected calibration error appear only as front-matter metrics. It is Apache-2.0 and English-only, built for Slack churn decisions. Packaging, not evidence.
bit-jev-2b-distilled is the cautionary case. It distils Kev 9B candidate logits at temperature 2.0 into a BitNet b1.58 2B backbone over 3,144 steps and two epochs, then adds a float32 pointer head. The file is about 1.19 GB and runs on CPU, but the card publishes no accuracy, Brier, NLL, or ECE, and has no separate open-weight licence.
Data mixtures and split hygiene
The mixture decides what the student can generalise to, and the split decides whether you can tell.
| Pack | Size | Teacher | Splits | Main caution |
|---|---|---|---|---|
| Claude-labelled distillation | 37,932 states, 150,827 questions | 27B Clef, clef-flash, Qwen3.5-27B, Opus re-judge | Train and validation | Author's protocol; synthetic states; 16,809 questions at label -1 |
| oss-jev-distill | 10,044 rows | jev-1.13.0 | 7,909 train / 1,074 eval / 1,061 calib | No schema or student results |
| Train-Jev 27B | 5,123 steps at batch 128 | SargeDev corpus | Single training run | Single seed; calibration in-distribution only |
| bit-jev 2B | 3,144 steps, 2 epochs | Kev 9B logits, T=2.0 | Single dev case | No accuracy; licence gap |
The Claude-labelled pack also shows language balance: roughly 11,203 English, 9,005 Korean, 8,889 Japanese, and 8,835 Chinese train states, with question and option language matching the state. The language-judgement pool is 19,025 records and is 55% English. If your deployment is outside that mixture, the student's calibration will not transfer just because the teacher's did.
Split hygiene is the cheapest quality gate. Keeping eval and calib out of training is only meaningful if you also keep them out of prompt selection and threshold fitting. The Claude-labelled validation set comes from the same generators, so it measures in-distribution generalisation, not a clean out-of-distribution test. Neither pack gives you a production holdout.
Student size and architecture
Student size buys capacity, not automatically accuracy. The 4B Train-Jev reports 0.647 PK accuracy, ECE 0.076, and a Decision Index of 40.4 on a 24 GB GPU at 43 ms p50. The 27B reports 0.851 transfer accuracy, ECE 0.053, and a Decision Index of 54.6 on an 80 GB GPU at 82 ms p50. Both use the same 5,123-step batch-128 recipe and 24-slot head, so the comparison is clean, and still single-seed.
At the small end, bit-jev's 2B student targets CPU and low memory, with a 1.19 GB file and a float32 pointer head. Its card reports 1,954 ms average on an EPYC 9654 at 32 threads for a 703-token, 77-candidate dev question, and 86.56 ms on an RTX 5090: latency, not accuracy. FunDM shows a smaller head over a frozen backbone can retain almost all of the full model's accuracy, but that is a different claim from a smaller student matching a larger one on your task.
Architecture matters as much as parameter count. A pointer head can score any option boundary, a 24-slot head cannot exceed its trained label set, and an encoder classifier like open-jev's DeBERTa build is capped at 512 tokens. A student that matches its teacher on short multiple-choice items can still fail on long states or large label spaces.
A recipe checklist
Start by identifying the teacher and version. "Jev" is not enough; oss-jev names jev-1.13.0, and a version change can move the labels. Demand soft targets and a documented split. Check whether the student trained on eval or calibration rows, and whether teacher labels were filtered for label -1 or disputes.
Compare student sizes at the same data and step budget, not across cards. Measure teacher agreement and gold accuracy separately, because a student can reproduce calibration while missing the task. Audit the licence chain: teacher terms, data terms, and student-weight terms are three different things.
The fine-tuning and calibration guide covers fitting the student after distillation, the open decision-model roundup places these recipes beside the rest of the field, and what is actually on Hugging Face covers the hub side. Read distillation and KL divergence before choosing a loss, and treat calibration as a property you have to re-measure after every student swap.
