ASSAY-001 calibrated Jev on CLINC150 at an expected calibration error of 0.0204, then found the same model systematically overconfident on Banking77 at 0.0936. Two domains, one audit, two verdicts — stated confidence is never a property of the model alone. What you are trusting is the pair: the weights and the domain you point them at. The practical response is a ledger: measure your domain's record as outcomes arrive, discount stated confidence to what it has earned, and sign the evidence.
Calibration belongs to a pair
Three independent studies, one shape.
| Study | In-domain result | Out-of-domain result |
|---|---|---|
| ASSAY-001 | CLINC150 ECE 0.0204 | Banking77 ECE 0.0936, overconfident |
| Does Jev know when it doesn't know? | OpenBookQA 94.2% accuracy at ECE 0.024 | Unseen priority rule 44.7% at ECE 0.325 |
| A pre-registered adversarial run | Routing ECE 0.075 | 3-SAT answered satisfiable at every clause ratio |
ASSAY-001 froze its protocol on 2026-09-17 before any query and logged zero type errors across 8,576 responses, so its Banking77 gap is about probabilities, not schema. The OOD study spent about $0.06 sending 3,721 public benchmark items and 900 rule-generated support tickets through the Vercel AI Gateway with zero data retention — and found the sign of the error flipping by question type: the unseen priority rule refit to T 1.92, still overconfident, its boolean question to 0.66. Calibration is per question shape as much as per domain.
Keep a record in your own domain
jev-trust treats the domain as the unit of trust. It logs every call to append-only JSONL with the question, answer, stated and effective confidence; record_outcome() builds your domain's record — accuracy, Brier score, ECE, calibration currency C = 1 − ECE — and effective_confidence reports what a stated number is worth where you run, from bin-observed, C-adjusted, or insufficient-n bases. Overconfidence alerts fire when a decision is confident enough to act on but the domain has not earned it. The PyPI page states no licence, and its supporting numbers are the author's linked Assay research — hosted Jev 1.13 at ECE 0.041 on closed deterministic tasks, and 50% accuracy while stating 91% confidence on synthetic email triage.
AnthusAI's sentiment study took 8,801 unique examples and cut raw ECE for P(positive) from 0.117 to 0.052 with Platt scaling, and to 0.008 with isotonic regression on the held-out split — isotonic matched or beat Platt at every calibration size from 20 to 5,280 examples. It is one constructed sentiment set on jev-1.13.0 with arbitrary neutral labels, so treat the magnitudes as the study's, not yours. The direction is why the independent evals lead with calibration; see how the benchmarks measured Jev.
Discount the stated number
AnyJev shows the discount in full. A zero-label temperature fix cuts ECE from 0.240 to 0.095; at a 5% error budget, auto-decidable answers rise from 7.7% to 52.0% with 100 to 500 labels; and its L2 head is a closed-form solve over 100 to 300 labels, finishing in seconds and adding roughly 100 KB at serving time. The limits matter: no transfer to another question set or model, vLLM needed on a GPU, fp8 costs accuracy.
Mapika's Decider Chat Gemma-4-31B makes the point with no training at all — stock Gemma-4-31B-it weights, a fixed answer slot, and a fitted temperature function T(n) = max(0.05, 10.124 − 1.633 ln n). It scores 57.33 on Decision Index v0.2.1 at ECE 0.047, behind Jev 1.13.0's 57.91, with the base model's behaviour unchanged.
Discounting also moves control flow: in jev-eval-agent, a 100-tool benchmark, confidence-gated routing blocks a reply when Jev chooses respond_to_user but done falls below JEV_DONE_THRESHOLD (0.5), exposing the next-best tool.
Sign the evidence
A ledger only settles arguments others can recompute. ASSAY-001 logs every request and response verbatim, seals them with SHA-256, and excludes smoke-test runs from scoring. The pre-registered evaluation indexes every number in its prose: verify_citations.py exits non-zero if a figure cannot be traced to a measurement, and its raw logs — 123,805 requests, 138 minutes, $12.69, five failures, all on jev-1.13.0 — ship as 1.3 GB compressed with zstd to 47 MB, with sha256 manifests. The OOD study publishes every raw Gateway response. A claim that cannot be recomputed is not an audit; it is a testimonial.
What the scoreboards add
JevBench freezes 534 cases per complete entrant — including 220 hard decisions written by Claude Opus 5 and GPT-5.6 Sol and hashed before any system ran — and scores the geometric mean of chance-corrected intelligence, calibration, speed and cost at 25% each. Jev 1.13.0 sits second at 74.4 behind Hopper's 75.4 on the 66-row board, at 950 input tokens and $0.0399 per 1,000 decisions. Caveat: held-out cases were exposed to the services measured.
SemIf reads typed option probabilities straight from a frozen Qwen3.5-4B's logits in one forward pass: on one RTX 3090, 21 binary criteria answered in 1.023 s against 5.332 s and 111 output tokens for an autoregressive JSON array, with shared mode prefilling one long state and branching it across criteria. Scores depend on the quantized weights. Nautilus Compass applies the same discipline to memory, reporting P@1 0.890 against mem0's 0.774 on LongMemEval-S and a drift detector at held-out AUC 0.83, p95 under 50 ms and a 0.5% production fire rate, from the authors' own numbers.
The ledger is the deliverable
Three habits compose. Measure the pair, not the model. Discount before you act — effective confidence comes from your outcomes, not a model card. Sign what you measured. None of the corrections transfers on its own: AnyJev's head is the proof, isotonic regression only earned its 0.008 inside one distribution, and the calibration record itself is per-domain. A gate tuned on someone else's traffic blocks work your domain handles fine or waves through the rule it gets wrong.
If you are setting those gates now, pair confidence gates in production with fine-tuning and calibration — the ledger is what tells you when the gate has drifted.
