What problem does the paper identify in System One models?

For choice questions, the answer depends on the order of the options and increasingly so as options grow in number, contradicting the promise that the same input returns the same answer in one forward pass.

Does averaging over option orders fix the problem?

Cyclic averaging reduces the order dependence without removing it: it still changes 9.3% of answers under shuffling in the two-model, five-dataset test and up to 15.4% when wrapped around eight models.

How does Runoff work?

It first scores each option alone, which order cannot affect, to keep a few candidates, then compares the candidates in listings where each takes every position. It needs no training and can run in two forward passes.

When does option order change the answer most?

Scoring each option alone shows the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident.

How accurate and fast is Runoff with many options?

With 20 options it is about as accurate as the alternatives and three to four times faster, according to the abstract.

Does Runoff need training or tuning?

It needs no training, and the paper reports that wrapped around eight System One models it removes the order dependence of the five benchmarks.

Is the Runoff answer order-independent by construction?

Yes. The paper states Runoff's answer does not depend on the option order by construction, because single-option scoring is order-free and candidates take every position in the comparison.

CommunityJev Practices & Patterns

Consider, Then Choose: Order-Invariant Decisions in System One Models (arXiv 2610.18233)

A study of option-order sensitivity in System One models and a training-free fix called Runoff. It scores each option alone to keep a few candidates, then compares them with each candidate in every position, removing order dependence in two forward passes.

See highlights

Open arxiv.org

Format: Article
Published by: Community
Last verified: Oct 2026

Tagscommunity research arxiv robustness inference evaluation

Works forJev for Classification Jev for Ranking

Used inJev for Software & DevTools Jev for Data & Analytics

Highlights

Finds System One choice answers depend on option order, increasingly so as options grow in number.
Averaging over cyclic shifts reduces but does not remove this dependence.
Scoring options alone shows order changes answers mostly when the model is unsure among its top few.
Runoff scores each option alone, keeps a few candidates, then compares them in every position.
On two open models and five datasets it changes no answer under shuffling, versus 9.3% for cyclic averaging.

Watch out

arXiv preprint posted October 13, 2026 (cs.AI, cs.CL) by Jianguo Huang, Yao Long Teng, Yanchen Deng, and Bo An. The results cover two open models, five datasets, and eight wrapped System One models; no hosted API or production pipeline is tested, and accuracy is reported as roughly on par rather than higher.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It (arXiv 2609.26758)

An evaluation of Jev and two open-weight typed decision models when option names and definitions conflict. With yes/no names, decision-flip rates are up to 70.4 percentage points higher than the 0/1 control, and mean AUC drops from 93.8% to a below-chance 23.2%.