Jev glossary
Evaluation & TrainingSigmoid calibration
Platt scaling
A two-parameter sigmoid fit on labeled data that maps a model's raw probability onto a better-calibrated one.
- Category
- Evaluation & Training
- Also known as
- Sigmoid calibration
- Reading time
- 1 min
- Related terms
- 4
- Directory entries
- 2
- Docs
- Not listed
- Added
- 2026-09-28
Definition
On 8,801 sentiment examples, Jev's raw expected calibration error for P(positive) was 0.117. A Platt curve fit on a calibration split cut that to 0.052 on held-out data. Isotonic regression on the same split reached 0.008, because the error was not sigmoid-shaped.
The AITA study used a related idea: a logistic regression learned from Jev's answers on 300 older posts, then adjusted probabilities on the 770 test posts. That chart is not a like-for-like win unless every model is adjusted the same way.
Tagscalibrationevaluation
Related terms
Definitions that connect to this one.
Isotonic regressionA non-decreasing fit from raw confidence to observed accuracy. It can follow a miscalibration that is not a sigmoid.Evaluation & TrainingCalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceBrier scoreA score for a full probability vector against one observed label. Lower is better, and a confident wrong answer costs more than a hedge.Evaluation & TrainingConfidenceA 0-1 number on every Choice and Score answer that collapses the probability distribution into one certainty score.Answers & Confidence
From the Jev directory
Directory entries that go deeper on this term.
A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPythonA 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonFrom the Jev AI model community
Posts from builders shipping with Jev right now.