Option-label bias
The tendency of a typed decision model to answer from an option's label rather than the definition attached to it, so renaming options to A and B can raise accuracy.
- Category
- Evaluation & Training
- Also known as
- Label-over-definition bias
- Related terms
- 4
- Directory entries
- 3
- Docs
- —
- Added
- 2026-10-14
Definition
A typed question gives every option a short label and a written definition, and the definition is where the developer states the rule the model should apply. Option-label bias is the measured failure mode where the probability follows the label instead: deleting every definition left one open model's accuracy unchanged at 0.8559 against 0.8487, and renaming options to A and B raised it by 0.1511.
The effect is prompt-rendering, not the constrained decision head: one code base writes each option as label plus definition, another writes only the definition, and changing that one expression made three checkpoints exactly invariant while creating the bias in the model that had none. The practical mitigation is to test a two-call probe where a label contradicts its definition, and to prefer definitions that restate the rule in the label's own words.
Related terms
Definitions that connect to this one.
From the Jev directory
From the Jev community
Posts from builders shipping with Jev right now.