Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Evaluation & TrainingData driftCovariate shift

Distribution shift

When the traffic a model sees at run time stops looking like the data it was calibrated on, so its probabilities no longer mean what they did.

Open docs.typesafe.aiAll 164 terms
Category
Evaluation & Training
Also known as
Data drift, Covariate shift
Related terms
5
Directory entries
56
Docs
docs.typesafe.ai
Added
2026-10-12

Definition

Distribution shift is the gap between the data a model was trained and calibrated on and the data it meets in production. A new product, a policy change, a language mix, or a seasonal spike can move the inputs enough that a previously correct threshold is now wrong.

It is the reason thresholds do not transfer across tasks and why calibration is a recurring task rather than a one-off. Watch for it by tracking the input mix and the confidence histogram over time, and re-check calibration whenever either moves.

Tagsevaluationbenchmarks

Related terms

Definitions that connect to this one.

CalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceBenchmark claimsTypeSafe's headline multipliers — 193.6x faster, 444.6x cheaper, a 238x lower input price — come from its own workflow comparisons.Evaluation & TrainingTemperature scalingDividing a model's logits by one fitted number per setting to bring its probabilities closer to observed frequencies.Evaluation & TrainingRepeatabilityHow stable answers are across repeated evaluations, measured separately from accuracy and reported as probability standard deviation.Evaluation & TrainingReliability diagramThe plot that bins predictions by confidence and compares each bin's average probability to the observed rate of correct answers.Answers & Confidence

From the Jev directory

Jevals — Independent Jev Benchmark for Models That Decide
An independent benchmark of hosted Jev against six LLMs on PubMedQA, Banking77, and HelpSteer2, scoring decisions against human labels and reporting per-board results for each model.
Jev Practices & PatternsArticle#community#benchmarks#evaluation
Community
Confidence
How TypeSafe reports certainty, how it differs from probability, and how to use it architecturally to gate and route decisions.
Jev Practices & PatternsDocs#official#confidence#routing
Official
Best Decision Model APIs in 2026
A ranked comparison of thirteen decision-model APIs as of October 2026, including Jev, OpenAI Decisions, Clef, Liquid d1, Kev, Perplexity Decider, and Strands, scored on evidence, usage, and price.
Jev Benchmarks & EvaluationsArticle#community#comparison#decision-model
Community
A decision model d1 surpasses Jev
GIGAZINE's report on Liquid AI's d1, which it describes as the first decision model to beat Jev on Hugging Face's Decision Index, and a short survey of the clone wave that followed Jev's September launch.
Jev Media & TalksArticle#community#news#benchmarks
Community
Typed Decisions Benchmark
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
Typed Decision Leaderboard
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community

50 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai