Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Answers & ConfidenceCalibration curve

Reliability diagram

The plot that bins predictions by confidence and compares each bin's average probability to the observed rate of correct answers.

Open docs.typesafe.aiAll 164 terms
Category
Answers & Confidence
Also known as
Calibration curve
Related terms
5
Directory entries
67
Docs
docs.typesafe.ai
Added
2026-10-12

Definition

A reliability diagram puts predicted probability on one axis and outcome frequency on the other, one point per confidence bin. A perfectly calibrated model lies on the diagonal: the cases it called 0.8 are right about 80% of the time. Deviations above and below the line show over- and under-confidence.

It summarizes calibration over groups of predictions, not single answers, and it is the picture behind the numeric scores like ECE and the Brier score. A diagram that looks good can still hide a bucket that is badly off, so read the bins, not only the line.

Tagsconfidenceevaluation

Related terms

Definitions that connect to this one.

CalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceBrier scoreA score for a full probability vector against one observed label. Lower is better, and a confident wrong answer costs more than a hedge.Evaluation & TrainingExpected calibration errorThe average gap between predicted confidence and observed accuracy, grouped into bins. Lower is better, and the number depends on the bins.Evaluation & TrainingProper scoring ruleA reward or error measure that is optimized only by reporting true probabilities — the training signal behind RLCD.Evaluation & TrainingConfidenceA 0-1 number on every Choice and Score answer that collapses the probability distribution into one certainty score.Answers & Confidence

From the Jev directory

Confidence
How TypeSafe reports certainty, how it differs from probability, and how to use it architecturally to gate and route decisions.
Jev Practices & PatternsDocs#official#confidence#routing
Official
AI Primer: RLCD and Calibration
Why TypeSafe trains decision models with RLCD instead of RLHF: calibrated probabilities where 0.2 outcomes happen about 20% of the time, and the case for machine-to-machine automation.
Jev Guides & ArticlesDocs#official#docs#evaluation
Official
Laya Typed-Decisions
The fine-tuned Laya checkpoint for four typed-decision workflows: 0.766 accuracy on 2,000 held-out decisions, above Jev 1.13.0's published 0.727 and the teacher self-agreement ceiling it trained past.
Jev Models & Reimplementations#community#open-models#local
Community
Decider Chat Gemma-4-31B
An untrained readout: stock Gemma-4-31B weights plus a decider config that reads option letters at the answer slot, scored #2 of 70 on Decision Index v0.2.1 at 57.33 against Jev 1.13.0's 57.91.
Jev Models & Reimplementations#community#open-models#local
Community
Typed Decisions Benchmark
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
Tasksource Jev Typed-Decisions
2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community

61 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai