Jev glossary
Evaluation & TrainingIsotonic calibration
Isotonic regression
A non-decreasing fit from raw confidence to observed accuracy. It can follow a miscalibration that is not a sigmoid.
- Category
- Evaluation & Training
- Also known as
- Isotonic calibration
- Reading time
- 1 min
- Related terms
- 4
- Directory entries
- 1
- Docs
- Not listed
- Added
- 2026-09-28
Definition
The sentiment study fit isotonic regression on a calibration split and measured expected calibration error of 0.008 on the held-out test, against 0.117 raw and 0.052 for Platt. It matched or beat Platt at every calibration size from 20 examples to 5,280.
After that fit, question wording moved raw error from 0.064 to 0.160 across 11 setups, and the calibrated error stayed between 0.006 and 0.018. The fit is for one dataset and jev-1.13.0. A new label set needs its own.
Tagscalibrationevaluation
Related terms
Definitions that connect to this one.
Platt scalingA two-parameter sigmoid fit on labeled data that maps a model's raw probability onto a better-calibrated one.Evaluation & TrainingCalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceConfidenceA 0-1 number on every Choice and Score answer that collapses the probability distribution into one certainty score.Answers & ConfidenceThresholdThe value your code compares against a score, a Noul probability, or a confidence to turn an answer into an action.Answers & Confidence
From the Jev directory
Directory entries that go deeper on this term.
A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPythonFrom the Jev AI model community
Posts from builders shipping with Jev right now.