What makes ProbabilityBench's ground truth exact?

Each environment's full data-generating process is explicitly defined, so the authors can analytically compute exact class probabilities and Bayes error rather than estimating them from single observed labels.

Which domains does the benchmark cover?

Three simulated text-based domains: peer review, covert information transfer, and credit risk.

How does it preserve probabilities when cases become text?

The paper provides a theoretical guarantee that rendering the cases into natural-language memos strictly preserves their underlying generative probabilities.

How are reported probabilities scored?

The exactness enables direct measurement of reported probability quality via KL divergence against the known per-instance class probabilities.

Which models sit on the cost-quality frontier?

Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider.

Can the benchmark evaluate calibration metrics themselves?

Yes. It serves as a meta-evaluation framework, using known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators.

What is the main limitation of the ProbabilityBench setup?

All three environments are simulated and their probabilities are known by construction, so the benchmark tests probability reporting quality rather than uncertainty on real-world data.

CommunityJev Benchmarks & Evaluations

ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty (arXiv 2610.17016)

A benchmark providing exact instance-level class probabilities across three simulated text domains: peer review, covert information transfer, and credit risk. Because the data-generating process is defined, exact probabilities and Bayes error are computable, enabling KL-divergence scoring of reported probabilities.

See highlights

Open arxiv.org

Format: Article
Published by: Community
Last verified: Oct 2026

Tagscommunity research arxiv calibration benchmark uncertainty

Works forJev for Scoring Jev for Evaluation

Used inJev for Finance & Fintech Jev for Security & Privacy Jev for Education

Highlights

Provides exact, instance-level class probabilities in three simulated text-based domains.
Domains are peer review, covert information transfer, and credit risk.
Defining the full data-generating process allows analytic computation of Bayes error.
A theoretical guarantee says natural-language memo rendering strictly preserves the generative probabilities.
Across 16 evaluated models, most on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider.

Watch out

arXiv preprint posted October 12, 2026 (cs.LG, cs.AI, stat.ML) by Yuyang Dai, Yuting Tang, Futoshi Futami, and Takashi Ishida. The three environments are simulated rather than drawn from production, and their exactness holds by construction, so the benchmark does not measure uncertainty on naturally occurring text.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration (arXiv 2609.35342)

Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.