Expected calibration error is a useful number and a bad summary. It collapses a whole distribution of predictions into one average gap, and that average depends on how you bin, how many labels you have, and what the base rate is. Two models with the same ECE can fail in opposite directions. One is underconfident on easy cases and overconfident on rare ones; the other is uniformly slightly off. The number cannot tell you which.
The better question is what you need the probability to do. If it feeds a threshold, you need the bin around that threshold to be trustworthy. If it feeds a weighted score, you need the whole distribution to be close in a proper-scoring sense. If it feeds a human review queue, you need to know how often the review actually changes the answer.
What ECE collapses
ECE groups predictions into bins, compares mean confidence with observed accuracy in each bin, and averages the gaps weighted by bin size. It is a diagnostic, not a proper scoring rule. A model that predicts the base rate everywhere can have low ECE and almost no resolution. A model that separates classes well can have a higher ECE because its confident bins are slightly overconfident.
TypedBench makes this concrete by judging calibration against the finite-sample noise floor of a matched, perfectly calibrated predictor rather than against an absolute target. With a few hundred labels, some apparent miscalibration is just sampling noise. The paper reports accuracy as a median and range across paraphrases, which is the right shape for a diagnostic: a range tells you whether the bin result is stable across question wording.
| Metric | Question it answers | What it can hide |
|---|---|---|
| ECE | How far average confidence is from accuracy, by bin | Resolution, bin choice, segment mix |
| Reliability diagram | Where the misses are | Small-bin noise, base rate |
| Brier score | How close probabilities are to outcomes | Whether errors are calibration or resolution |
| Log loss | How costly confident mistakes are | Indifference to useful discrimination |
| AUROC | How well probabilities rank positives | Absolute probability placement |
Reliability bins are a diagnostic
A reliability diagram is more useful than its summary because you can see which bins are populated. If most cases land in the top bin, lower bins tell you little. A twelve-case bin is not an estimate to tune on.
TypedBench's hosted model follows the stated policy but is systematically underconfident. An underconfident model makes a fixed threshold escalate more often than the error rate justifies, which raises cost without buying accuracy. Under asymmetric costs, the paper reports that using the hosted model's probabilities can be worse than taking its top answer, because the probability is not placed where the cost matrix needs it.
TypeSafe's confidence guide separates confidence from probability. Confidence is a 0–1 statistic derived from the distribution; Noul answers carry no confidence at all. The page's three-option demo approximates it as (3 × largest probability − 1) / 2, and the example routes anything below 0.5 to a human while requiring above 0.9 to auto-approve a transfer. That is a policy choice, not a calibrated probability. The full distribution still comes back, so you can compute your own statistic.
Brier and base rates
The Brier score is a proper scoring rule: it rewards both calibration and resolution. A rough decomposition separates reliability, resolution and uncertainty. The uncertainty term is set by the base rate, so the same predictions score differently across prevalences. A model calibrated on a 5% positive rate is not automatically calibrated on a 50% rate, even if the question text is identical.
Beyond Answer Confidence shows why this is not only a prevalence problem. With no answer-relevant information, Jev assigned up to 0.80 to a salient option. On news beyond an observed knowledge boundary, confidence exceeded accuracy by 0.21–0.33, a gap that recalibration on earlier months did not close. Whether the evidence suffices read at AUROC 0.95, against 0.85 for confidence on the same items. Asking whether Jev knows the answer flagged fabricated entities at 0.91, but showed no advantage with realistic names or removed dates.
That is the distinction between calibration and self-knowledge. A model can be calibrated on familiar closed-choice tasks and still fail as an indicator of missing knowledge. The AI primer states the narrow claim correctly: across many predictions, outcomes assigned 0.2 should occur about 20% of the time. It is a property of groups, not a promise about one answer.
Repeatability is not calibration
A model can return the same probability every time and still be wrong. TypeSafe's Choice consistency cookbook ran 15 repeats across 9 conditions over 8 Choice questions. Jev's mean probability standard deviation was 0.0098, against 0.0245 to 0.0543 for five LLM distributions. With top probability below 0.60 routed to uncertain, policy agreement reached 99.2% while 74.2% of labels stayed automatic. Jev still flipped its top label on primary_risk and link_handling across its own repeats.
The Noul consistency cookbook ran 14 Noul questions over 15 repeats. Mean probability standard deviation was 0.0102 at 111 ms per rubric call. The covered answers spanned 0.43 to 0.53, crossing the 0.5 threshold, while the other 13 stayed on one side. A 0.30–0.70 band maps those middle values to uncertain for human review with no extra call. The cookbook calls the band illustrative rather than calibrated, and warns that values near its edges can still flip.
The shared caveat matters: these are repeatability numbers, not accuracy numbers, and a deterministic temperature-0 baseline can score 100% agreement by never abstaining. Low variance stabilizes a gate; it does not set it correctly.
Personas and languages move the target
Calibrated to Whom put 288,000 answers to the 24 Values Survey Module items through Jev. Answers were highly repeatable, with an ICC of 0.997, and without a persona they resembled Jev's own American personas. A Saudi persona moved answers toward the human Saudi–US difference, reproducing 87% of its size in English but 62% in Arabic. A language cross showed the smaller Arabic difference came from the language of the items, not the language of the persona description. Age shifted profiles about as much as nationality, and Jev was less confident in Arabic and for Saudi personas.
For calibration, that means the target is conditional. A probability threshold fitted on English, no-persona traffic is a different operating point from the same threshold on Arabic persona traffic. The jaggedness notes add the structural version of the same warning: complementary probabilities can sum past 1, and the same question can disagree across primitives. The probability-axioms test found negation pairs miss summing to one by 0.064 and three labels sum to 1.14. These are coherence failures, but they land in the same threshold.
A calibration protocol
- Fit on labels from your own prevalence, not a balanced benchmark.
- Report reliability bins with counts and confidence intervals, not just ECE.
- Report Brier and its decomposition alongside the diagram.
- Test the knowledge boundary with items just beyond the model's observed range.
- Test segments separately: language, persona, document type, customer tier.
- Treat confidence as a derived statistic and version the threshold with the question text.
- Keep an abstention band for values near the decision boundary, and measure the review rate it creates.
The practical link to production is confidence gates. If you are recalibrating rather than gating, fine-tuning and calibration covers the trade-offs, and what the papers found puts the calibration results beside the coherence results. The expected calibration error, Brier score and base rate entries define the terms.
