A typed decision is supposed to depend on the state and the question. Three things that are neither still move the answer: the order of the options, the names attached to them, and how many options there are. The effects are measurable, they grow with option count, and they are large enough to change a production verdict.
The useful response is not to declare the model unreliable. It is to treat the option set as part of the question, test the transformations that should be neutral, and decide which ones you are willing to let change the answer.
Order is an input
Consider, Then Choose tested option-order sensitivity across two open System One models, five datasets and eight wrapped models. Choice answers depended on the order of the options, and the dependence grew as the list grew. Averaging over cyclic shifts reduced the effect without removing it: answers still changed 9.3% of the time under shuffling, and up to 15.4% when the model was wrapped around eight candidates.
The proposed fix, Runoff, scores each option alone (a comparison that order cannot affect), keeps a few candidates, then compares the survivors with each candidate in every position. It needs no training and can run in two forward passes. With 20 options it was about as accurate as the alternatives and three to four times faster.
Kyle Wild's order-bias test reports the opposite for hosted Jev: it was not tricked by option ordering in his tests, while several other decision models were. That is a useful counterexample, but the post's chart data and other-model names were not recoverable from the catalog digest. Treat it as one reproduction, not a general law.
The larger pre-registered test is harder to dismiss. The IIA study used 299,463 calls to jev-1.13.0 on BANKING77, CLINC150 and AG News. Every added option shifted the log-odds between the correct label and its strongest rival by 0.20 to 0.52 nats RMS, about 1.2 to 3 times run-to-run variation. The distortion grew with the log of the option count, reaching roughly one nat at 255 options. Questions packed into one call did not interact, with bias below 0.001.
Labels carry polarity
Type-Safe Is Not Error-Free changed only the name-definition mapping, leaving the question, state, and the name and definition texts unchanged. With yes/no names, decision-flip rates were up to 70.4 percentage points higher than with the 0/1 control across all four binary rules. Reassignment with yes/no names lowered mean AUC from 93.8% to a below-chance 23.2%. Random strings as option names produced flip rates close to neutral controls across all three models, and the type-error rate stayed 0% throughout.
That last detail matters: type-correct output is not definition-correct output. A model can return a valid option every time and still follow the name rather than the definition bound to it. TypedBench found the hosted model follows the stated policy but is wording-sensitive and systematically underconfident. The lesson is to keep labels short, distinct and neutral, and to put the real distinction in the criteria rather than in a name that carries its own polarity.
Option count and irrelevant alternatives
The IIA result is not a uniform penalty. An option similar to the wrong answer draws probability from it, shifting about +0.10 nats toward the gold label. A paraphrase of the wrong answer yields an intermediate pattern. Off-domain padding pushes the odds against the gold label. So the direction depends on what the added option looks like, which is why a fixed "option-count correction" is unlikely to work.
Scoring each option with an independent yes/no question removes the option-set dependence, but the same study reports it picks the right answer 8.7 points less often than the hosted model. That is the trade: independence buys stability at the cost of accuracy. The practical middle ground is to keep the list small, make near-miss options deliberate, and test the list you actually ship.
A table of levers
| Lever | Measured effect | Mitigation |
|---|---|---|
| Option order | 9.3% of answers change under shuffling; up to 15.4% wrapped | Freeze order; use Runoff for high-stakes choices |
| Option names | Up to 70.4 pp higher flip rate with yes/no names | Neutral labels; definitions carry the distinction |
| Option count | RMS 0.20–0.52 nats; about 1 nat at 255 options | Keep lists short; avoid off-domain padding |
| Wording | Paraphrase range in TypedBench | Report median and range, not one run |
| Irrelevant alternatives | Directional shift from similar options | Test near-misses explicitly |
A different mitigation is to average over framings rather than over option orders. Jev Consensus Gate asks each proposed action three independent questions (safety, legitimacy, and adversarial framing) and requires consensus. All three allow passes; any single block blocks; any escalate goes to a human. Three calls cost roughly $0.00006 per action at $0.042 per million input tokens. It is MIT licensed, Python 3.12, and ships the action, perspective and consensus modules.
That pattern spends more calls to catch a framing blind spot. It is worth it when a wrong allow is expensive and a wrong block is cheap. It is not a substitute for testing order and labels, because all three calls can share the same blind spot if the question text is the same.
Metamorphic testing is the method
JevBench Metamorphic packages the idea as a benchmark. It uses 240 synthetic English cases, 20 in each of 12 decision domains, with 2,240 questions: 1,341 Nouls, 615 Choices and 284 Scores. Fifty metamorphic relations in five dimensions edit only the question side and are frozen in advance. The full run executes 12,000 tests and 10,320 requests; the mini run 1,200 tests and 1,248 requests. The rewrite bank holds 823 accepted rewrites from 1,697 candidates, including paraphrases, option descriptions, Chinese translations and negations.
Its caveat is the one to carry into your own suite: coherence is not correctness. A model that answers uniformly scores higher than every evaluated model, so a coherence score has to be read beside accuracy and the Brier score. A model can be perfectly consistent and consistently wrong.
You do not need the full benchmark to use the method. Take 50 labeled cases from your own queue, apply five transformations (option order, label rename, paraphrase, one irrelevant option, one near-miss) and log the full distribution. The cases that flip are your question-design backlog.
The design rule
The jaggedness notes put literal reading and structural invariants on the same list as counting and dates. The model answers the words you wrote, not the intent behind them. That makes the option set and the question text first-class inputs.
- Freeze the option order in production and test whether shuffling changes the answer.
- Use neutral, distinct labels; do not rely on a definition to beat a misleading name.
- Keep the option list short and make near-misses deliberate.
- Report a paraphrase range, not a single accuracy number.
- Use a two-pass or consensus design when a framing error is expensive.
- Version the question text and option list alongside the model version.
The broader treatment is in question design, wording, order and labels. For the option-set side, see candidate coverage and irrelevant alternatives, and for the failure modes that are not framing effects, designing around jagged edges. The option order bias, option label bias and probability coherence entries define the terms.