What property does the probability-axioms paper test?

Coherence: whether probabilities given to logically related questions fit together item by item, which a system that acts on those probabilities needs.

Why does the coherence test need no labels?

It uses a battery of logically linked questions, such as asking whether the label is X, is not X, is one of the other two, and which label applies, so correctness labels are unnecessary.

What data and scale does the coherence test use?

160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels, yielding 480 negation pairs to check.

How incoherent is Jev according to the tests?

Its probabilities for a label being X and not X miss summing to one by 0.064 on average, with a 95% CI of 0.055 to 0.072, and its three single-label probabilities sum to 1.14.

How does Qwen3.8-27B compare on coherence?

It misses by 0.293 with first-token probabilities and 0.122 with verbalized probabilities, and rejects both a statement and its negation in 196 of 480 pairs.

Where do the coherence violations concentrate?

Differently for each system: Qwen's first-token readout does not become more coherent where it is confident, while Jev's violations concentrate where its answer is uncertain and run about five times its repeat noise.

Where are the coherence code and data?

They are available at github.com/bro789/typed-decision-coherence.

CommunityJev Benchmarks & Evaluations

Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms? (arXiv 2609.33209)

Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.

See highlights

Open arxiv.org bro789/typed-decision-coherence

Stars: 0
Language: Python
License: MIT
Format: Article
Published by: Community
Last verified: Oct 2026

bro789/typed-decision-coherence

User repository on GitHub

View on GitHub

Data and analysis code for 'Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?'

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity research arxiv calibration logic evaluation

Works forJev for Verification Jev for Evaluation

Used inJev for Software & DevTools Jev for Healthcare & Life Sciences

Languages: Python

Highlights

Asks whether the label is X, whether it is not X, whether it is one of the other two, and which label applies.
Uses 160 items from ChaosNLI and PubMedQA, each with three mutually exclusive labels.
Jev's negation pairs miss summing to one by 0.064 on average (95% CI 0.055 to 0.072).
Qwen3.8-27B misses by 0.293 with first-token probabilities and 0.122 with verbalized probabilities.
Jev's three single-label probabilities sum to 1.14 on average, and its violations are about five times its repeat noise.

Watch out

arXiv preprint posted September 27, 2026 (cs.CL, cs.AI, cs.LG; 17 pages) by Keyi Li, Yihao He, and Quanyi Li. The coherence battery is label-free but limited to 160 items and two datasets, and the abstract notes the two systems fail in different ways, so the gap is not a single capability ranking.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring (arXiv 2609.33538)

A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.