Skip to content
Evaluation & TrainingLabel-over-definition bias

Option-label bias

The tendency of a typed decision model to answer from an option's label rather than the definition attached to it, so renaming options to A and B can raise accuracy.

Category
Evaluation & Training
Also known as
Label-over-definition bias
Related terms
4
Directory entries
3
Docs
—
Added
2026-10-14

Definition

A typed question gives every option a short label and a written definition, and the definition is where the developer states the rule the model should apply. Option-label bias is the measured failure mode where the probability follows the label instead: deleting every definition left one open model's accuracy unchanged at 0.8559 against 0.8487, and renaming options to A and B raised it by 0.1511.

The effect is prompt-rendering, not the constrained decision head: one code base writes each option as label plus definition, another writes only the definition, and changing that one expression made three checkpoints exactly invariant while creating the bias in the model that had none. The practical mitigation is to test a two-call probe where a label contradicts its definition, and to prefer definitions that restate the rule in the label's own words.

Tagsevaluationcalibrationresearch

From the Jev directory

An analysis of open-weight typed decision models that finds their probabilities mostly follow option labels rather than written definitions. Deleting every definition leaves laya-td accuracy unchanged (0.8559 versus 0.8487), while renaming options to A and B raises accuracy by 0.1511.
Jev Practices & PatternsArticle#community#research#arxiv
Community
An evaluation of Jev and two open-weight typed decision models when option names and definitions conflict. With yes/no names, decision-flip rates are up to 70.4 percentage points higher than the 0/1 control, and mean AUC drops from 93.8% to a below-chance 23.2%.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai