A question is code. It changes behavior when the wording changes, it can regress when the model changes, and it should be reviewed when the policy changes. Most teams treat it like configuration (a string in a file, edited when someone remembers) until a small wording change quietly moves a production gate.
A question registry fixes that by giving every question an identity, a version, and a regression suite. The registry does not have to be a platform. A table in the repository is enough, as long as the workflow reads from it and the tests run against it.
Start with the identity
Every question needs a stable ID and a version. The ID names the judgment, not the wording: refund.precedence.final_sale, ticket.routing.category, document.extraction.total_amount. The version increments when the wording, options, or criteria change.
The workflow evals show why end-to-end matters. It scores four workflows (security incidents, agent trace observability, invoice processing, and customer service) as workflows of Noul, Choice, and Score questions, not as single prompts. Jev averaged 67.8% accuracy at $0.0004 and 0.4 seconds per case across the four. The code behind it runs 150 invoice cases, 204 customer-service cases, 111 agent traces, and 240 security incidents against typesafe:jev-1.13.0.
The caveat is important: the workflow code is assumed correct, and the reference labels come from two frontier models. That is fine for comparing models. It is not a substitute for your own labeled set when the question is whether your policy is right.
Store the question with its evidence
A registry entry should include:
| Field | Why it matters |
|---|---|
| Question ID and version | Lets you trace a production answer to the exact wording |
| Primitive and options | Defines the answer shape |
| Prompt text and criteria | The thing being versioned |
| State fields it reads | Tells you what can invalidate the question |
| Owner and review date | Makes drift someone's job |
| Test set | The cases that must not regress |
The jaggedness notes are a useful source for the test set. Add cases for counting, dates, literal reading, and contradictory criteria. The TypedBench method (paraphrases, calibration against a noise floor, and asymmetric costs) gives you a way to structure the suite without building a research harness.
Run the suite like a regression test
A question registry is only useful if it runs. For each question, keep 20–100 labeled cases and a small set of perturbations:
- The original phrasing.
- Two paraphrases.
- The option list in a different order.
- One near-miss option.
- One out-of-scope option.
Record the answer, the probability of the correct label, and the confidence. A version bump should come with a diff of those numbers. If a wording change improves one case and breaks three, the diff tells you before the deploy does.
The shadow and canary article covers the production side. Shadow mode gives you real traffic against the old and new question versions; the registry gives you the cases to explain what changed. Log the state hash, the question version, the confidence, and the policy version together.
Version the model separately
A question can regress because the model changed, not because the wording did. The SDK changelogs and JavaScript changelogs are the release notes for the client, not the model, but the principle is the same: pin what you can and record what moved.
jev-latest is an alias. The models guide and the API reference define the model field and the pinned names. Store the pinned model version next to the question version. A test result that says “question v3 passed” is incomplete if it does not say which model answered.
The reproducibility article goes deeper on the pinning stack. For the registry, the minimum is two columns: question version and model version.
Review the registry on a schedule
A registry that nobody reviews becomes a museum. Put a review date on every entry and a quarterly review on the calendar. At review time, ask three questions:
- Does the question still express the policy?
- Do the test cases still represent the traffic?
- Has the model or the option set changed enough to invalidate the threshold?
The confidence docs are the reference for the threshold question. Re-run the calibration slice when the model changes. The workflow evals show the end-to-end effect, which can be larger than the question-level effect.
Start with the questions that can hurt you
You do not need to register every question on day one. Start with the ones that gate an irreversible action, feed a customer-visible message, or have already changed once without a test. Those are the entries where a wording edit is most expensive.
A registry of five high-risk questions with real test cases is more useful than a registry of fifty prompts with no owner. Expand it as the workflow grows, and let the review date (not the next incident) decide when an entry gets attention.
A short checklist
- Give every question a stable ID and a version.
- Store the wording, options, criteria, state fields, owner, and review date.
- Keep a labeled test set with paraphrases, option-order changes, and near-misses.
- Run the suite on every version bump and record the diff.
- Pin the model version alongside the question version.
- Log state hash, question version, confidence, and policy version in production.
- Review the registry on a schedule, not only when something breaks.
The registry is the cheapest way to make question changes boring. Boring is what you want when a wording edit can move a production gate.
