What problem does the paper study for recalibration?

Many related prediction problems, which the authors call questions, each come with few labels. The paper asks what information can be shared across them and what is necessary for good decisions, with compound risk measured as excess Brier score over the true per-question maps.

What is the minimax compound risk the paper derives?

(mn)^-2/3 plus min{τ²,1/n} plus min{τ²,V²,(V/n)^{2/3}} up to universal constants, for maps within sup-distance τ ≤ 1 and total variation V ≤ 2 of one nondecreasing shape, for every non-atomic output law.

Which procedure attains the minimax rate?

A pooled isotonic fit with a per-question correction sized once on held-out labels of all questions attains it without knowing τ, V, or the output law, for n ≥ 2 labels per question and m ≥ 3 questions.

What is lost by tuning each question separately?

Tuning each question on its own labels loses a factor of order min{n, log(m²/n)} when the maps differ by constants and m²/n is large, according to the paper's analysis.

What does the identification limit say with random effects?

When the maps are drawn from an unknown prior, other questions need not identify the Bayes rule. The identification regret is at most 2^-n once the law of a question's data is known, extending a 1968 bound of Skibinsky for coins.

What is β* and what role does it play?

β* is about 3.583. The worst-case identification regret is at least β*^-n/(72√n) at every finite m; for constant maps the exponential base is exactly β*, not the 4 suggested by the moment range, and for monotone maps it lies in [2, β*].

Who should read this recalibration paper?

Readers building recalibration or shared-label pipelines across many low-label questions, and anyone interested in the limits of pooling. The paper treats decision models such as Jev as the motivating application.

CommunityJev Practices & Patterns

Recalibrating Many Questions with Few Labels: Minimax Rates and Identification Limits (arXiv 2610.18488)

A theoretical study of recalibrating many related yes/no questions that each have few labels, motivated by decision models such as Jev. It derives minimax compound-risk rates and shows pooled isotonic fitting with one per-question correction achieves them, while tuning questions separately loses a factor.

See highlights

Open arxiv.org

Format: Article
Published by: Community
Last verified: Oct 2026

Tagscommunity research arxiv calibration theory

Works forJev for Scoring Jev for Evaluation

Used inJev for Software & DevTools Jev for Data & Analytics

Highlights

Motivated by decision models such as Jev, which return a probability for any yes/no question a developer writes.
Gives minimax compound risk (mn)^-2/3 + min{τ²,1/n} + min{τ²,V²,(V/n)^{2/3}} for maps near one shape.
A pooled isotonic fit with a single per-question correction attains it without knowing τ, V, or μ.
Tuning each question on its own labels loses a factor of order min{n, log(m²/n)} when maps differ by constants.
Identification regret is at most 2^-n once a question's data law is known; worst case at least β*^-n/(72√n).

Watch out

arXiv preprint posted October 13, 2026 (cs.LG, stat.ML, 69 pages) by Xiaoyu Li and Chiwun Yang. The results are theoretical, stated up to universal constants for non-atomic output laws, so they do not establish how these rates behave on a real hosted model or with discrete probability outputs.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models (arXiv 2609.38827)

An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.