Skip to content
Evaluation & TrainingCoherence

Probability coherence

Whether a model's probabilities for logically linked questions fit together item by item: a label and its negation should sum to one, and single-label probabilities should not exceed it.

Category
Evaluation & Training
Also known as
Coherence
Related terms
4
Directory entries
1
Docs
—
Added
2026-10-14

Definition

Calibration checks each probability against outcomes; coherence checks the probabilities against each other. Asking whether a label is X, whether it is not X, and whether it is one of the other labels needs no ground-truth labels, and Jev's negation pairs missed summing to one by 0.064 on average while its three single-label probabilities summed to 1.14.

The distinction matters because a system that acts on probabilities consumes the relationships, not just the numbers. Jev's violations concentrated where its answers were uncertain and ran about five times its repeat noise, and the comparison model failed differently, rejecting both a statement and its negation in 196 of 480 pairs. Because the checks need no labels, they can run before deployment on question forms you actually use.

Tagsevaluationcalibrationresearch

From the Jev directory

Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai