A typed decision looks deterministic: one state in, one probability distribution out. In practice, the same judgment moves when you rename an option, reorder the list, or wrap the question in a different sentence. That movement is not random noise. It is a property of the model you have to design around, measure, and sometimes correct in code.
The useful question is not whether wording matters. It does. The useful question is which wording choices you are willing to let change the answer, and which ones you need to hold fixed.
Order is a real input
Consider, Then Choose tested option-order sensitivity across two open System One models, five datasets, and eight wrapped models. It found that choice answers depend on the order of the options, and the dependence grows as the option list grows. Averaging over cyclic shifts reduced the effect without removing it: answers still changed 9.3% of the time under shuffling, and up to 15.4% when the model was wrapped around eight candidates.
Kyle Wild's order-bias test reports the opposite for hosted Jev; it was not tricked by option ordering in his tests, while several other decision models were. That is a useful counterexample, but note the caveat: the post's chart data and other-model names were not recoverable from the catalog digest, so treat it as one reproduction, not a general law. The right response is to test your own model and question set rather than importing either conclusion.
A pre-registered study of irrelevant alternatives makes the same point with a much larger sample. The IIA test used 299,463 calls to jev-1.13.0 on intent and topic classification. Every added option shifted the log-odds between the correct label and its strongest rival by 0.20 to 0.52 nats RMS, about 1.2 to 3 times run-to-run variation. The distortion grew with the log of the option count, reaching roughly one nat at 255 options. Options that look similar to the rival pull probability away from it; off-domain options hurt the correct label.
The design consequence is simple: the option set is part of the question, not a neutral menu.
Labels are part of the state
TypedBench evaluated a hosted model, an open encoder, and open decoders from 0.8B to 9B parameters on the same items. It reports accuracy as a median across paraphrases and calibration against a finite-sample noise floor. The hosted model followed the stated policy but was wording-sensitive and systematically underconfident. That second finding matters for gates: if the model is underconfident, a threshold tuned on raw probabilities will escalate more than the error rate justifies.
Option labels carry their own weight. Type-Safe Is Not Error-Free finds that typed decision models can follow the option name rather than the definition bound to it. If billing_issue and payment_problem both appear, the model may prefer one because of its surface form, not because your definition says it is the better match. Write definitions, but do not assume they dominate the name.
Two habits follow. Keep labels short, distinct, and meaningful on their own. And when two options are genuinely close, make the distinction explicit in the criteria rather than relying on a hidden definition the model may not weight.
Measure wording, not just accuracy
A single accuracy number hides the instability that matters in production. TypedBench's paraphrase range is a better starting point: run each item under several phrasings and report the spread, not only the mean. The IIA test's option-set perturbation is another: add plausible near-misses and watch whether the correct label's probability moves for reasons unrelated to the evidence.
Metamorphic coherence is the same idea in benchmark form. It checks whether a decision model gives consistent answers to transformations that should not change the answer. You do not need a published benchmark to copy the method. Take 50 labeled cases from your own queue, apply five transformations (option order, label rename, paraphrase, one irrelevant option, one near-miss) and log the answers. The cases that flip are your question-design backlog.
For high-stakes choices, a two-pass approach can reduce order dependence without retraining. The Runoff paper's method scores each option alone, keeps a few candidates, then compares the survivors with each candidate in every position. It removed shuffling changes in the paper's tests while keeping accuracy roughly on par. That is a useful pattern when a wrong answer is expensive and an extra forward pass is cheap.
Write the question for the decision, not the demo
The primitives guide and the advanced structure guide both push toward small, specific questions. A question that asks the model to compare two policies, extract a date, and decide whether to escalate in one sentence gives you no way to tell which part moved. Split the judgment into atomic questions, then combine the answers in code where the policy is visible.
The jaggedness notes are the checklist for the parts the model does not do reliably: counting, date arithmetic, exact magnitude between score levels, and structural invariants. Do not ask a question whose answer depends on one of those. Ask for the component, then calculate.
A short checklist
- Freeze the option order for production requests, and test whether shuffling changes the answer.
- Use short, distinct option labels; do not rely on definitions to beat a misleading name.
- Run a paraphrase spread on 50–100 labeled cases before trusting a threshold.
- Add near-miss options to the test set and watch the correct label's probability, not only the top answer.
- Use a presence Noul or a code check for anything the model reads literally.
- Version the question text alongside the model version, because both are inputs.
None of this makes the model deterministic. It makes the instability visible, which is the part you can actually manage.