Probability coherence
Whether a model's probabilities for logically linked questions fit together item by item: a label and its negation should sum to one, and single-label probabilities should not exceed it.
- Category
- Evaluation & Training
- Also known as
- Coherence
- Related terms
- 4
- Directory entries
- 1
- Docs
- —
- Added
- 2026-10-14
Definition
Calibration checks each probability against outcomes; coherence checks the probabilities against each other. Asking whether a label is X, whether it is not X, and whether it is one of the other labels needs no ground-truth labels, and Jev's negation pairs missed summing to one by 0.064 on average while its three single-label probabilities summed to 1.14.
The distinction matters because a system that acts on probabilities consumes the relationships, not just the numbers. Jev's violations concentrated where its answers were uncertain and ran about five times its repeat noise, and the comparison model failed differently, rejecting both a statement and its negation in 196 of 480 pairs. Because the checks need no labels, they can run before deployment on question forms you actually use.
Related terms
Definitions that connect to this one.
From the Jev directory
From the Jev community
Posts from builders shipping with Jev right now.