A probability can be calibrated and still be incoherent. Calibration asks whether outcomes assigned 0.7 happen about 70% of the time across many predictions. Coherence asks whether the numbers for logically related questions fit together on a single item: whether a label and its negation sum to one, whether three mutually exclusive labels sum to one, whether adding an irrelevant question leaves the others alone.
Those are different tests, and passing one does not imply passing the other. The first needs labels and a population. The second needs only the model's own answers to linked questions.
Calibration and coherence are different tests
The probability-axioms paper is the clearest statement of the gap. It asks whether Jev's probabilities satisfy coherence using logically linked questions, so no correctness labels are needed. A model can be well calibrated on a benchmark and still give a label 0.8 and its negation 0.4. The first number can be right on average; the pair is still impossible.
TypedBench tests calibration against a finite-sample noise floor and reports accuracy as a median and range across paraphrases. That is a strong test of probability placement, but it does not test whether the probabilities for the same item obey the axioms. The confidence guide adds another distinction: confidence is a derived 0–1 statistic, and Noul answers carry no confidence property at all. A gate built on confidence is not the same as a gate built on the full distribution.
| Test | What it uses | What it asks | What it misses |
|---|---|---|---|
| ECE / reliability bins | Labels and bins | Is average confidence close to accuracy? | Logical relations, resolution |
| Brier / log loss | Labels | Are probabilities close to outcomes? | Pairwise coherence |
| Negation and sum tests | Linked questions, no labels | Do probabilities obey the axioms? | Correctness |
| Paraphrase range | Rewrites | Is the answer stable under wording? | Logical implication |
| Exact-probability simulation | Known data-generating process | How close is the whole distribution? | Natural text |
The negation test
The paper uses 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, yielding 480 negation pairs. It asks X, not-X, one-of-the-other-two, and which label applies.
Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average, with a 95% confidence interval of 0.055 to 0.072. Its three single-label probabilities sum to 1.14 on average. The violations are about five times its repeat noise and concentrate where its answer is uncertain. Qwen3.8-27B misses by 0.293 with first-token probabilities and 0.122 with verbalized probabilities, and rejects both a statement and its negation in 196 of 480 pairs. The two systems fail differently, which is why the authors do not read the gap as a single capability ranking.
The test is label-free, but it is not free of limits. It covers 160 items and two datasets, and the abstract says the gap between the systems is not a single capability ordering. Treat the effect size as a reason to run the test on your own questions, not as a universal constant.
Conjunction and framing
The bias audit found a related failure on conjunction tasks. Across 15,000 responses on 30 conjunction tasks, Jev violated the underlying probability rule in 54% of responses, with accuracy varying dramatically across formulations and answer orders, even though it answered a well-known probability problem correctly every time. At the level of selected answers, the audit found Jev largely avoided the characteristic choice patterns of several human biases. The probabilities still moved.
That is the practical warning. A dashboard that checks only the top answer will miss incoherence that a threshold reads directly. The audit also found sensitivity to irrelevant numerical information and to option ordering, and cognitive-bias manipulations shifted returned probabilities even when the selected answer stayed the same.
Exact ground truth: ProbabilityBench
ProbabilityBench takes a different route to the same question. It defines three simulated text domains (peer review, covert information transfer, and credit risk) with a fully specified data-generating process. That makes exact instance-level class probabilities and Bayes error computable, and a theoretical guarantee says rendering the cases into natural-language memos preserves the generative probabilities. Reported probabilities can then be scored by KL divergence against the known values.
Across 16 evaluated models, most on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. The benchmark also serves as a meta-evaluation: because the ground truth is known, it can test how accurate standard calibration metrics and Bayes-error estimators are. Its main limitation is built in. All three environments are simulated and exact by construction, so it measures probability reporting quality rather than uncertainty on naturally occurring text.
What coherence failures do to gates
The jaggedness notes list the structural version of the problem: complementary probabilities can sum past 1, and the same question can disagree across primitives. Those are not edge cases you can patch with a better prompt. They are reasons to validate the numbers you gate on.
Beyond Answer Confidence shows why a coherent-looking probability can still be the wrong signal. On news beyond an observed knowledge boundary, confidence exceeded accuracy by 0.21–0.33, and recalibration on earlier months did not close the gap. Whether the evidence suffices read at AUROC 0.95, against 0.85 for confidence on the same items. Asking whether Jev knows the answer flagged fabricated entities at 0.91 but showed no advantage with realistic names or removed dates.
A gate treating a high probability as license to act assumes two things: that the probability is placed correctly, and that it is about the right thing. Calibration evidence supports the first only within its measured population. Coherence evidence tests the second only for the relations you wrote down.
The evidence audit and the training claim
The early evidence audit reviewed 28 papers posted in the nine days after Jev's release and concluded that the typed readout had not shown an independent accuracy advantage over comparable label-probability readouts. Its clearest gains were latency and cost, with accuracy gaps remaining on harder tasks. It derives a 14-item evaluation checklist from recurring weaknesses in the first wave of studies. Coherence and probability placement belong on that checklist, not only accuracy.
The AI primer explains the training target behind the claim: RLCD trains calibrated probabilities rather than preferences, and calibration is a property of groups of predictions. It also names mode dropping as a risk of preference optimization. None of that guarantees item-level coherence. A model trained to produce good marginal probabilities can still assign impossible joint values.
How to test coherence on your questions
- Write a negation pair for every binary question and check whether the two probabilities sum to one.
- For a three-way choice, sum the three single-label probabilities and flag deviations.
- Add a conjunction case where the joint probability cannot exceed either component.
- Ask the same question alone and in a batch, and compare the full distribution.
- Paraphrase the question and the options, and compare the full distribution, not only the top label.
- Use a simulated domain with known probabilities if you need an absolute check.
- Log probabilities, not only answers, so the failure is visible before it reaches a threshold.
The broader pattern is the one in designing around jagged edges: ask for the component the model can answer, then enforce the logic in code. What the papers found puts the coherence results beside the calibration results, and measuring Jev covers the field reports. The probability coherence, structural invariants and proper scoring rule entries define the terms.
