Two calls with the same state and the same question can return different probabilities. Sometimes the difference is small; sometimes it flips the label. If a production gate depends on that number, “the model said so” is not a reproducibility story. You need to know which inputs are pinned, which ones move, and how much variance the decision can tolerate.
Reproducibility is not accuracy. A model can be consistently wrong, and a perfectly reproducible answer can still be a bad policy. But without reproducibility, you cannot tell a regression from noise.
Pin the model
jev-latest moves by design. The models guide defines the model field and the aliases; the API reference is the canonical list. A gate tuned against one release can behave differently against the next, and the change may not announce itself in your code.
The Jev 1.14 launch post is a concrete example. jev-latest moved to 1.14.0 about a week after launch. Older versions stay available with notice: jev-1.13.0 had 60 days before formal deprecation, and jev-1.14.0 gets 30 days. That is enough time to test, but only if you know which version you are on.
Pin the version in configuration, not in a prompt. Store it next to the question version and the policy version. The SDK changelogs and JavaScript changelogs are useful for the client side; the model version is separate.
Pin the question
The question registry article covers the mechanics. For reproducibility, the key is that the question text, options, criteria, and legends are part of the input. A wording change is a model change from the answer's point of view.
TypedBench found that a hosted model followed the stated policy but was wording-sensitive and systematically underconfident. That means a paraphrase can move both the label and the confidence. If you do not version the wording, you cannot tell whether a shift came from the model or from the edit.
The IIA study adds option-set dependence. Adding an option shifts the log-odds between the correct label and its strongest rival by 0.20 to 0.52 nats RMS. The option set is part of the question version.
Log the state, not just the answer
A decision is reproducible only if you can replay the state. Log a hash of the state, the full question payload, the model version, the raw probabilities, the threshold, and the policy version. The shadow and canary article recommends the same set for rollout logs. The confidence-gates article adds the question version and the policy version explicitly.
Hashing the state is a trade-off. A hash is cheap and privacy-preserving, but it does not let you replay the case without the original payload. For regulated workflows, store the payload under the same retention rules as the decision. The data retention article covers the policy side.
Measure variance before you trust a threshold
The self-consistency cookbook is the template. It runs the same rubric 15 times per condition and reports Jev's mean probability standard deviation at 0.0098, against 0.0245 to 0.0543 for the LLM distributions it compares. Routing anything below 0.60 probability to an uncertain bucket lifted policy agreement from 90.8% to 99.2%, while still labeling 74.2% of cases automatically.
The lesson is not the exact numbers. It is that variance is measurable, and a gate should be set with that variance in view. If the model's probability moves by 0.02 between repeats, a threshold at 0.60 is effectively a band from 0.58 to 0.62. Treat it that way.
The jaggedness notes add the structural warning: complementary probabilities can sum past one, and the same judgment can disagree across primitives. Do not assume a single run is a clean sample from a stable distribution.
Separate model drift from policy drift
A change in outcomes can come from four places:
- The state changed.
- The question changed.
- The model changed.
- The threshold or policy changed.
Logging all four together is the only way to tell them apart. The workflow evals show end-to-end behavior; the question registry shows question-level diffs; the model guide shows version changes. Use them as separate signals rather than one dashboard that blends everything.
For a production review, replay a fixed sample of cases through the old and new versions. Report the disagreement rate and the confidence shift. If the disagreement is concentrated in a slice (a language, a document type, a customer tier), you have found a distribution change, not noise.
Reproducible does not mean identical
A reproducible decision pipeline can still return different probabilities on two runs. What it gives you is the ability to say why: the same state, the same question version, the same model version, and a measured variance band. That is enough to distinguish a regression from noise.
Do not promise bit-for-bit identity. Promise a replayable record and a known tolerance. The tolerance is a property of your workflow, not a claim about the model.
A short checklist
- Pin the model version and the question version together.
- Log state hash, question payload, probabilities, threshold, and policy version.
- Run the same cases repeatedly and record the variance before setting a threshold.
- Treat a threshold as a band, not a point, when variance is non-trivial.
- Replay a fixed sample when the model or question changes.
- Keep model releases and policy releases as separate review events.
- Report disagreement and confidence shift, not only accuracy.
Reproducibility is the ability to explain a difference. Once you can explain it, you can decide whether it is a regression, a distribution shift, or just the model being a model.