Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Evaluation & TrainingOrder sensitivityOrder-flip

Option order bias

The change in a model's answers when the same options are presented in a different order, measured as an order-flip rate on reversed or rotated candidate sets.

All 172 terms
Category
Evaluation & Training
Also known as
Order sensitivity, Order-flip
Related terms
4
Directory entries
3
Docs
—
Added
2026-10-14

Definition

If a question is a judgment about the state, the order of the options should not change the answer. Order bias measures the degree to which it does: one open model changed 30% of its answers when the option order was reversed, and pre-registered tests of hosted models found option-set and order effects that survive into production thresholds.

The mitigations are cheap. Averaging over cyclic rotations lowers order-flip rates and raised accuracy on all eleven models one paper tested, and order-invariant scoring schemes that consider the options jointly rather than one at a time remove the dependence at the readout. Test with the same state and a shuffled criteria list before trusting a routing decision.

Tagsevaluationcalibrationresearch

Related terms

Definitions that connect to this one.

Option-label biasThe tendency of a typed decision model to answer from an option's label rather than the definition attached to it, so renaming options to A and B can raise accuracy.Evaluation & TrainingOption markerScoring each answer option at its own marker token, then softmaxing over the question's options so the answer space is defined at request time.API & ModelsLiteral readingJev answers the question you wrote, not the one you meant; scoping words, negations, and implied conditions are read at face value.Evaluation & TrainingCalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & Confidence

From the Jev directory

Consider, Then Choose: Order-Invariant Decisions in System One Models (arXiv 2610.18233)
A study of option-order sensitivity in System One models and a training-free fix called Runoff. It scores each option alone to keep a few candidates, then compares them with each candidate in every position, removing order dependence in two forward passes.
Jev Practices & PatternsArticle#community#research#arxiv
Community
More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models (arXiv 2609.38827)
4GitHub stars
An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.
Jev Practices & PatternsArticle#community#research#arxiv
CommunityGlax147/jev_ordinal_scale_biaPython
Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev (arXiv 2610.16655)
A pre-registered study of whether jev-1.13.0 satisfies independence of irrelevant alternatives, using 299,463 calls on intent and topic classification. Every added option shifts the log-odds between the gold and strongest wrong label by 0.20–0.52 nats RMS, 1.2 to 3 times run-to-run variation.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai