A decision model in a live system is usually a replacement for something: a rule, a heuristic, a smaller model. Replacing it in one step is how you discover that your thresholds were tuned on the wrong data. The safer path has three stages — shadow, canary, and rollback — and each one has a specific thing it is trying to learn.
Shadow mode: run it beside the incumbent, act on neither extra
In shadow mode, every request that reaches the incumbent also reaches Jev, and Jev's answer is logged but never acted on. The point is to collect real distributions — how confident Jev is on your traffic, where it disagrees with the incumbent — without any user-visible risk. The confidence docs frame this as the way to choose a threshold from data rather than guesswork.
The subtlety: because output is free and the state is already assembled, shadowing is cheap. You are paying only input tokens for the second opinion, which makes a long shadow period affordable where it would not be for a generative model.
Pick the threshold from the disagreement, not the accuracy
The interesting number in shadow mode is not "how often Jev is right" — you often do not have labels — but the shape of the disagreement. Where the incumbent says yes and Jev says no with high confidence, you have a case to inspect. The confidence-gates article walks the threshold-setting process; the reliability question is whether Jev's confident answers are the ones you would have endorsed.
A decision model is well suited to this because it returns a probability, so you can bucket the shadow log by confidence and see the policy at every candidate threshold before you pick one.
Canary: act on a slice, watch the metric
Once you trust the threshold, act on a fraction of traffic — one percent, one queue, one region — and watch the outcome metric that motivated the change. This is where an ordinary A/B framework works, with one twist: version-pin the model. The jaggedness notes are explicit that jev-latest moves under you, so a canary and its control must be pinned to a version, or the comparison drifts mid-test.
The Katalon testing guide makes the same recommendation for QA: pin the model version for regression tests, because an alias update can change results with no code change on your side.
Rollback has to be one flag
A canary you cannot reverse quickly is not a canary. The decision call should sit behind a single switch that reverts to the incumbent instantly, and the switch should be the same one shadow mode used. If reverting means a deploy, you will hesitate at the exact moment you should not.
The workflow evals entry is the companion: you cannot roll back what you cannot measure, so the metric that decides rollback has to exist before the canary starts.
What to log for all three stages
- The full state hash, so a case can be replayed.
- The questions, so a disagreement can be traced to a question.
- The raw probabilities and the threshold applied.
- The model version, pinned.
- Which stage produced the decision — shadow, canary, or live.
The independent evaluations and the measuring Jev article agree on the core finding: question design dominates, so the shadow and canary logs are really a record of how well your questions were written, and reading them is how you improve the questions before you widen the rollout.
Shadow to learn the distribution, canary to learn the outcome, pin the version, and keep rollback to one flag. The three stages exist to make each irreversible step small.