A Choice can only return one of the options it was given. That single constraint means every typed question needs an answer for "none of these" — and the research of the past month says that answer is the hardest one to get right. This is how to design for it: rejection options, candidate coverage, and what code should do when the probability lands in the middle.
Give the question a way out
The docs recommend an explicit other or none_of_the_above option whenever the list may not cover the input, and the probability on that option becomes your abstention signal. Without it, a state the taxonomy never anticipated still gets a confident pick from the least-wrong option, which is exactly the failure the arithmetic study measured.
Rejection is weaker than it looks
On paired arithmetic problems with an explicit rejection option, answer-present accuracy reached 99% while correct rejection fell to 7% — across magnitudes, operation depths, and rejection labels. The same model answered a direct Noul verification of candidate correctness at 99%. The gap is between verifying an option and choosing the rejection option, and the fix is unglamorous: a decision threshold selected on separate development problems raised rejection to 79% while keeping 97% answer-present accuracy.
Coverage is a separate number
Accuracy hides two failure sources: picking wrong from the list, and the list not containing the answer. The coverage benchmark separates them with artificial omissions, retrieval misses, and out-of-scope queries — and finds rejection policies do not transfer across tasks. A Jev policy calibrated on DBpedia rejected 69.3% of covered Emotion inputs; an Emotion policy lost detection entirely. Measure coverage per candidate set, not per model.
Decide what code does with a low probability
Three outcomes, three policies:
| Signal | Policy | When it fits |
|---|---|---|
| Rejection option wins | Route to fallback or re-ask with a wider list | Taxonomies that drift |
| Confidence below threshold | Escalate to a stronger model or human | Expensive errors |
| Confident but wrong | Shadow mode first | New question shapes |
The confidence-gated routing pattern is the documented version of the middle row, and the confidence gates article covers threshold selection. The key discipline is that the gate is code: the model returns the probability, and your threshold decides.
Escalation is the other side of rejection
The Laya reproduction study pre-registered a gate and found the frozen rule beat random escalation but missed its 10% accepted-set error target on both tracks — a useful reminder that a gate is only as good as its held-out evaluation. A System One transfer study found a 10–30% escalation band routed to a local classifier matched or beat the better of the two systems, and routing to an LLM instead matched that LLM at 24–77% lower API cost.
Test rejection before you trust it
- Include a rejection option in every list that can be incomplete, and gate on its probability.
- Build an out-of-scope test set — inputs you know are not in the taxonomy — and measure rejection on it, not only accuracy on covered inputs.
- Tune the threshold on one split and check it on another; the arithmetic study's 7%→79% swing came from exactly that discipline.
- Re-measure when the candidate set changes: option-set effects are real even for hosted models.
Rejection is not a fallback for a bad question. It is the mechanism that lets the rest of your taxonomy stay narrow and honest.
