Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Evaluation & TrainingECE

Expected calibration error

The average gap between predicted confidence and observed accuracy, grouped into bins. Lower is better, and the number depends on the bins.

All 164 terms
Category
Evaluation & Training
Also known as
ECE
Related terms
4
Directory entries
67
Docs
—
Added
2026-10-10

Definition

ECE is how open models are compared on honesty. AutoTrust reports 0.0009 for JEV-27B against the distilled corpus labels; opendecider-nano reports 0.092 next to Jev's 0.164 and Laya's 0.327 in its own table; the Typed Decision Leaderboard reports ECE alongside its AUC ranking.

Read it beside accuracy and provenance. The typed-decisions benchmark shows a very sharp distribution can score 0.727 accuracy with KL 1.442, and a vendor's ECE is measured on the vendor's bins — which is why the leaderboard publishes its grading code next to the scores.

Tagscalibrationevaluation

Related terms

Definitions that connect to this one.

CalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceBrier scoreA score for a full probability vector against one observed label. Lower is better, and a confident wrong answer costs more than a hedge.Evaluation & TrainingTemperature scalingDividing a model's logits by one fitted number per setting to bring its probabilities closer to observed frequencies.Evaluation & TrainingBenchmark claimsTypeSafe's headline multipliers — 193.6x faster, 444.6x cheaper, a 238x lower input price — come from its own workflow comparisons.Evaluation & Training

From the Jev directory

AutoTrust JEV-27B
An open distillation of Jev 1.13 onto Qwen3.8-27B: a 108.9M-parameter decision block answers typed questions in one prefill pass and adds a POST /v1/decide route beside a full System 2.
Jev Models & Reimplementations#community#open-models#local
Community
OpenDecider Nano
A 400M-parameter Jev-compatible decision model with a browser build: 0.796 on typed decisions against Laya's 0.766, 17 ms per question on an L40S, and an ONNX file that runs in a tab.
Jev Models & Reimplementations#community#open-models#local
Community
Typed Decision Leaderboard
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
Laya Typed-Decisions
The fine-tuned Laya checkpoint for four typed-decision workflows: 0.766 accuracy on 2,000 held-out decisions, above Jev 1.13.0's published 0.727 and the teacher self-agreement ceiling it trained past.
Jev Models & Reimplementations#community#open-models#local
Community
Decider Chat Gemma-4-31B
An untrained readout: stock Gemma-4-31B weights plus a decider config that reads option letters at the answer slot, scored #2 of 70 on Decision Index v0.2.1 at 57.33 against Jev 1.13.0's 57.91.
Jev Models & Reimplementations#community#open-models#local
Community
Typed Decisions Benchmark
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community

61 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai