Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Evaluation & TrainingKullback-Leibler divergenceRelative entropy

KL divergence

A directional measure of how far one probability distribution is from another, used to compare, distill, and monitor decision heads.

Open docs.typesafe.aiAll 164 terms
Category
Evaluation & Training
Also known as
Kullback-Leibler divergence, Relative entropy
Related terms
5
Directory entries
7
Docs
docs.typesafe.ai
Added
2026-10-12

Definition

KL divergence measures the extra cost of encoding samples from one distribution using another, and it is asymmetric: the divergence from A to B is not the divergence from B to A. In training it drives distillation, where the student matches the teacher's softened distribution rather than only the hard labels.

It shows up in evaluation as a way to compare two models' output distributions on the same states, and in monitoring as a drift signal when today's distribution moves away from the reference. Zero means identical; larger means more unlike.

Tagstrainingevaluation

Related terms

Definitions that connect to this one.

Cross-entropy lossThe training objective that penalizes a confident wrong probability monotonically, and the proper scoring rule behind log loss in evaluation.Evaluation & TrainingDistillationTraining a smaller model to reproduce a larger model's output distribution, including its probabilities rather than labels alone.Evaluation & TrainingTemperature scalingDividing a model's logits by one fitted number per setting to bring its probabilities closer to observed frequencies.Evaluation & TrainingLogit readoutReading the next-token probability of each declared label, then renormalising, so no answer text is generated.PatternsDistribution shiftWhen the traffic a model sees at run time stops looking like the data it was calibrated on, so its probabilities no longer mean what they did.Evaluation & Training

From the Jev directory

Jev Distill Corpus v3
The 741,000-row distillation corpus behind AutoTrust's Jev students: choice, score, and noul rows with full target distributions across yuri and openjev source streams, Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
AI Primer: RLCD and Calibration
Why TypeSafe trains decision models with RLCD instead of RLHF: calibrated probabilities where 0.2 outcomes happen about 20% of the time, and the case for machine-to-machine automation.
Jev Guides & ArticlesDocs#official#docs#evaluation
Official
Jebadiah
5GitHub stars
Open System One decision models (27B, 9B and 4B on Qwen chat bases, in bf16, GGUF and MLX) that answer Choice, Noul and Score questions with a probability for every option from one forward pass. The repo carries the trainer, data builders, evaluation harnesses and a record of every run.
Jev Models & Reimplementations#community#decision-models#open-weights
Communitygetainode/jebadiahPython
Luce
8GitHub stars
Open recipe for calibrated decision models: an LLM teacher writes training data from a one-sentence task description, a LoRA plus decision head on Qwen3-4B-Base answers Choice, Score and Noul questions with calibrated probabilities, evaluated next to Jev on identical items.
Jev Models & Reimplementations#community#calibration#training
Communityscienthoon/lucePython
tev1
227GitHub stars
Together AI's open recipe and weights for a Jev-inspired decision model fine-tuned on Qwen3.5-4B, with the full data pipeline, training config, and saved benchmark results.
Jev Repos & SDKs#community#python#open-models
Communitytogethercomputer/tev1Python
The Bitterest Lesson
A TypeSafe blog essay arguing that in ML the order that matters is doing the right task, then data, then compute, then algorithms, using the InstructGPT result as its example.
Jev Guides & ArticlesArticle#official#article#models
Official

1 more matching entry in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai