A backfill is a data pipeline with a model in the middle. The corpus is fixed, the questions are known, and the output is a table of probabilities. That makes it much easier than online serving, and much easier to get subtly wrong, because a bad batch can rewrite months of decisions before anyone notices.
The discipline is the same one you would apply to a warehouse migration: define the unit of work, measure the cost, version every input, checkpoint, and validate on a sample before running the whole thing.
The unit is a request, not a row
The natural mental model is one row per decision. The billing model does not work that way. Jev bills input tokens, and the state is sent once per request, not once per question. So the efficient unit is one request per item with all the questions that item needs.
The parallel questions cookbook makes the gap concrete. It ran 13 questions over a 54,000-character document: one batched call cost $0.000497 and took 0.27 s; 13 single-question calls cost $0.006090 and 2.71 s. The answers were the same across five repeats. That is 12.2x cheaper and 10x faster, and the saving is the shared state.
For a backfill, this changes the shape of the job. You are not running one model call per row. You are running one call per row with a question set, and the question set should cover every branch the downstream logic might take. The speculative fan-out pattern is the design for that: ask the category plus the four judgments the branch might need, then ignore what the branch does not use.
Measure the fixed overhead
The fan-out benchmark measured 2,976 requests totaling $0.156 and found about 261 input tokens of fixed overhead on every request. At eight questions, median input-token savings were 76% on a tiny control state, 77% on a small state, 84% on a medium state, and 86% on a large state.
That 261-token floor is the reason very small items should be grouped carefully. If each item is a one-line record and you send one request per record, overhead can dominate the bill. If each item is a long document, overhead is noise and the state is the whole cost.
The benchmark also found that 1,000 questions succeeded in one call, and that context window limits bracketed 32,768 content tokens and 65,536 total. Those are empirical limits from an independent run, not a vendor guarantee. The models page states a 64k context covering 32k of state plus the longest question, which is consistent but not a promise.
For a backfill, estimate before you run. Count the tokens in a representative item and question set, multiply by the number of items, and compare that to the per-item cost you are replacing. The benchmark's full run sent about 2 million input tokens in roughly 15 minutes for about $0.09 at the published $0.042 per million input tokens. A corpus of 100,000 items is a different conversation.
Chunk by context, not by file
The tempting chunk is the file. The right chunk is the unit a question can judge. A 200-page PDF split at arbitrary page boundaries produces states that are individually incomplete and collectively redundant. A document split at section boundaries, with only the relevant section passed to the question, produces smaller states and better answers.
The jaggedness notes add a specific warning: counting is unreliable and error grows with size. Do not ask one question for the total number of items in a long document. Ask one Noul per item and sum in code. That changes the batch design from one request per document to one request per document with a question per item, which is exactly the fan-out shape the billing model rewards.
The API reference caps Choice at 255 options and Score at 2 to 10 levels. If a backfill has a candidate set larger than 255, pre-filter in code or run multiple passes. Do not compress the options into a free-text label and hope the model returns the one you meant.
Version the job, not just the model
A backfill is only reproducible if you can name every input: the model version, the question text, the criteria, the threshold, the pre-processing code, and the state assembly. Pinning jev-1.13.0 is necessary but not sufficient. A question wording change can move answers more than a model patch.
jev-workbench is a useful reference for the versioning discipline. Publishing a judgment function freezes the config and model version, requires a successful preview of exactly that config, and never changes under a caller. Grants pin a client token to specific function versions, so a new publish does not silently upgrade anyone. Whether or not you use the tool, copy the rule: a backfill runs against an immutable function version, not against whatever is live.
The Jev harness shows the same idea inside an agent. Its context picker asks 22 yes/no questions in one call, one per runbook section, and its gate classifies a tool call as read, additive, modify, or destroy. Every decision is logged with its numbers. That log is what makes a replay possible after a bad run.
Checkpoint and make retries idempotent
Backfills fail in the middle. A 429, a malformed state, a single record that exceeds the context, a process restart: any of these can leave the corpus half-processed. The job needs a durable record of which items completed and which did not.
Use the item id as an idempotency key and write the result before advancing the checkpoint. The LLM cost optimizer does this with SQLite WAL, atomic reservations, durable audit leases, and idempotency keys, and it permits at most two attempts. That is a reasonable default for a backfill: retry once for a transient error, then quarantine the item for inspection rather than looping.
The API reference says to retry only 429 and 529, with exponential backoff. A 422 is a caller error; retrying it wastes time and hides the bad record. Classify the error before deciding whether the checkpoint advances.
Validate on a stratified sample
Do not run the full corpus to discover that the question set is wrong. Run a stratified sample first: a few items from each source, language, document length, and label. Compare the batch output to a hand-labelled subset, and inspect the disagreements rather than only the accuracy number.
The workflow evals code is a usable harness for this. It runs four workflows (150 invoice cases, 204 customer-service cases, 111 agent traces, and 240 security incidents) against typesafe:jev-1.13.0 or another provider, with inputs and labels from Hugging Face. It is Apache-2.0, needs Python 3.13 or newer, and its README is explicit that the workflow code is assumed correct: it compares models, it does not re-check the questions.
The published workflow evals report 67.8% average accuracy at $0.0004 per case across those workflows. That is a vendor-run number with reference labels from two frontier models. It is useful as a prior, not as a substitute for your own sample.
If the sample looks good, run the full corpus in shards and keep the sample results as a regression baseline. The statistical power article explains how many labelled cases you need before a difference is meaningful; the one request, many questions article explains why batching is the default. The reproducibility article is the checklist for the version fields.
A backfill is finished when you can rerun it and get the same table. That requires more than a saved model id. It requires a frozen question set, a checkpointed corpus, an idempotent writer, and a sample you trust.
