A router is a decision model with a different job: instead of answering the user, it answers “which path should this request take?” That makes it one of the most practical places to put a fast model, and one of the easiest to overbuild.
The useful pattern is a cascade, not a single classifier. Cheap deterministic checks first, then a small decision model, then a larger model or a human only when the earlier tiers cannot decide. The Hybrid Intent Router is a clean example: regex and tries, then TF-IDF with CatBoost, then System One models (Laya, Jev, CLM), then a generative LLM fallback. Every exit leaves a replayable decision record.
What a router actually decides
The intent-routing pattern asks two questions in one request: a Choice for the intent and a Score for complexity. Intent confidence below 0.5 routes to a human. order_status goes to deterministic code. Complaints escalate when complexity exceeds level 1 or its confidence falls below 0.5.
That is the right shape for most routers. One question selects the branch; a second question measures how hard the branch is. The confidence-gated routing pattern adds the per-action threshold: a balance check can act at 0.6, a transfer needs 0.85 plus confirmation.
The important design choice is that the router's output is a pair (answer and confidence), not a single label. Routing on the label alone lets a confidently misrouted request slip past review.
The cheap tiers come first
A decision model is fast, but a regex is faster. The Hybrid Intent Router puts regex and tries before any model, then a TF-IDF/CatBoost tier, then System One models, then an LLM. That ordering is not an accident: each tier is more expensive and more capable than the last, and the router only pays for the next tier when the current one cannot decide.
The S1 LLM Auto-Router shows how much a small model can do in one pass. It answers seven router questions (category, difficulty, stakes, and four flags) in a single ONNX pass. It is built on a pruned mmBERT-small, distilled from a fine-tuned teacher, with per-head calibration temperatures. The card reports about 14 ms p50 on four pinned EPYC cores and 26 ms on four Xeon cores, with 0.898 category accuracy and 0.819 all-seven-exact on 2,175 held-out synthetic requests across 22 languages.
Those numbers are self-reported, and the gold labels are a plurality vote of up to five LLMs, with an independent blind set of only 100 requests. Treat them as a strong starting point, not a settled benchmark. The design lesson is still sound: a small, purpose-built router can answer several routing questions in one pass for far less than a frontier call.
A router is not a model selector
There is a subtle difference between routing a request and selecting a model. A model selector chooses among providers by cost, latency, or capability. A router chooses a workflow branch: deterministic code, a specialist model, a larger model, or a human.
The Pi TypeSafe Router is a coding-agent extension that classifies each request and routes it to one of three configured models. It supports TypeSafe, Cloudflare, Vercel, and OpenRouter backends, includes doctor checks and shadow mode, and starts off until enabled. That is a model selector with a decision-model front end.
The System Switch paper is the research version of the same idea. It studies a fast actor that hands control to a reasoning model only when a gate opens. On 900 held-out questions, deferring the least confident 30% of decisions gained +0.13 with one option order, and +0.08 with shuffled options. The closed-loop result was negative (no variant reached the exit), which is a useful warning: a good offline deferral rule does not guarantee a good closed-loop policy.
Calibrate the gate, not just the router
A router's threshold is a policy. If the router says “route to the small model” at 0.7 confidence, the cost of a wrong route is a bad answer from the small model, and the cost of a missed route is an unnecessary large-model call. Those costs are asymmetric, and the threshold should reflect them.
The confidence notes are explicit that confidence is not accuracy. A router can be well calibrated on average and still be wrong on the slice that matters. Measure the routing agreement (how often the router's chosen branch matches the branch a human would choose) separately from the downstream answer quality.
The System Switch result adds a second measure: AUROC for predicting when the fast model will be wrong. Accuracy, calibration, and sensitivity are distinct. Two routers with the same accuracy can differ widely in how well they rank their own errors, and the ranking is what a deferral gate depends on.
Start with three branches
A first router does not need seven labels and four flags. Start with three branches: deterministic code, a decision model, and a human. Add a fourth only when you have a measured case for it.
The intent-routing pattern is a good template. Pick the intents that cover most of the traffic, add a none or other option for the tail, and route on confidence. The candidate coverage article covers the option-set design; the dependent questions article covers when a second request is better than a bigger router.
A short checklist
- Put deterministic checks before the model.
- Ask for intent and complexity in one request.
- Route on the answer and the confidence together.
- Set thresholds per branch, by the cost of a wrong route.
- Measure routing agreement separately from downstream quality.
- Report AUROC for error ranking, not only accuracy.
- Start with three branches and add more only with evidence.
- Leave a replayable decision record at every exit.
The router is the part of the agent that decides where to spend the next token. Get the cascade right, and the expensive model becomes the exception rather than the default.
