A decision model can only choose from the options you give it. That sounds obvious, and it is the most common source of silent failure in production. The state contains the right evidence, the question is clear, the model is reasonably calibrated, but the correct answer is not on the list.
Candidate coverage is the discipline of making sure the option set contains the labels the system actually needs, plus an honest way to say “none of these.” It is a policy problem before it is a model problem.
Coverage is not the same as accuracy
The rejection-policy benchmark separates candidate coverage from rejection behavior. It compares Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, and tests three different ways candidates go missing: artificial omission, natural retrieval misses, and public out-of-scope queries.
The result is the part to internalize. A Jev rejection policy calibrated on DBpedia rejected 69.3% of covered Emotion test inputs. An Emotion policy lost detection entirely when moved to another target. With ten intent candidates, Laya detected 99.0% of out-of-scope queries but rejected 48.8% of covered queries. In other words, a rejection threshold tuned on one candidate distribution does not preserve its operating point on another.
That is not a reason to avoid rejection options. It is a reason to treat coverage and rejection as two separate measurements. Coverage asks whether the right label was available. Rejection asks whether the system declined when it should have. A model can be good at one and bad at the other.
The option set is an input, not a menu
The IIA study makes the point experimentally. In 299,463 calls to jev-1.13.0, adding an option that was neither the correct answer nor its strongest rival still shifted the log-odds between those two by 0.20 to 0.52 nats RMS. The shift was directional: similar options drew probability from the rival, and off-domain padding hurt the correct label. At around 255 options, the distortion reached roughly one nat.
So “just add every possible label” is not neutral. It changes the answer, costs input tokens, and can make the correct label less likely. The advanced structure guide offers a better pattern for large taxonomies: pass a subtree per option so every candidate path stays available in one call, but trim each branch to its direct children plus a sample of leaves. The docs are explicit that untrimmed subtrees grow fast and inflate the bill.
Give the model a way to say no
A rejection option is a candidate label whose job is to absorb inputs that do not belong to any of the others. It is not a substitute for a confidence gate. The two work at different levels:
- The rejection option handles “none of these labels is right.”
- The confidence gate handles “I am not sure which of these is right.”
You need both when the cost of a false label is high. A rejection option without a gate can be chosen too rarely; a gate without a rejection option can only escalate to a human, which may be the wrong outcome if the input belongs to a category the system does not support.
The confidence-routing pattern shows the three-way branch: below a threshold, escalate; above it, act; for irreversible actions, require a higher threshold plus confirmation. Add a rejection option when the taxonomy is open, and measure the rejection rate against the covered and out-of-scope slices separately.
Build the candidate set from the domain
The most reliable candidate sets come from the work, not from a brainstorm. Pull a sample of real inputs, tag them with the decision each one needed, and group the tags into a first taxonomy. Then add the labels that the system must produce even if they are rare: “unknown sender,” “duplicate,” “already resolved,” “requires legal review.”
For retrieval-backed decisions, the candidate set has a second failure mode: the right label exists, but retrieval did not surface it. The rejection benchmark calls this natural retrieval miss. Log the retrieved candidates alongside the chosen answer so you can tell whether a wrong answer was a ranking problem or a classification problem. If the correct option was never retrieved, no threshold can save it.
The jaggedness notes add a useful warning: structural invariants do not hold, so do not assume probabilities over a candidate set behave like a clean distribution. Complementary probabilities can sum past one, and the same judgment can disagree across primitives. Treat the candidate set as a policy artifact, not a mathematical guarantee.
Measure the set, not only the model
A candidate-set evaluation needs at least four numbers:
| Measure | What it answers |
|---|---|
| Coverage | Was the correct label present? |
| Top-1 accuracy | Did the model choose it when present? |
| Rejection rate on covered inputs | How often does the system decline a case it could handle? |
| Rejection rate on out-of-scope inputs | How often does it decline a case it should decline? |
Report them on the same slices. A single “accuracy” number conflates a missing label with a model error, and those have different fixes.
TypedBench is a useful model here because it reports accuracy across paraphrases and calibration against a noise floor rather than a single point estimate. You do not need its full harness to copy the discipline. Run your candidate set against a held-out sample, perturb the options, and record both the answer and the probability of the correct label.
A short checklist
- Write down the candidate set as a versioned artifact.
- Include a rejection option when the taxonomy is open, and gate confidence separately.
- Test coverage and rejection on covered and out-of-scope slices.
- Add plausible near-misses to the evaluation set; watch how the correct label moves.
- Trim large taxonomies to the subtree the request needs.
- Log whether the correct option was retrieved before blaming the model.
- Re-calibrate the rejection threshold whenever the candidate set changes.
The goal is not a bigger menu. It is a menu whose gaps are known, measured, and handled by a policy you can defend.
