Jev answers every question in a request against the same state, and it answers them in parallel. That makes batching cheap, but it also creates a design trap: people write questions as if each one can depend on the answer to the previous one. It cannot. If a question needs the result of another, it has to go in a second request, or the dependency has to move into code.
This is the practical guide to that boundary: when to fan out, when to split, and how to keep the two-request patterns from turning into a chain of round trips.
Parallel means independent
The speculative fan-out pattern is the clearest statement of the contract. One request carries a Choice category plus four speculative Score and Noul judgments. All five are evaluated in parallel, and the code ignores the answers the taken branch does not use. Adding questions usually has little effect on response time. The cost is input tokens, because every speculative question sees the state on every call.
The parallel-questions cookbook makes the economics concrete. A 13-question regulatory briefing (8 Noul, 2 Choice, and 3 Score questions over a pinned ~54,000-character document) cost $0.000497 and took 0.27 seconds as one batched call. The same 13 questions as individual calls cost $0.006090 and 2.71 seconds. That is 12.2x cheaper and 10x faster, with the same answers across five repeats. The caveat is important: the latency gap shrinks if the individual calls run concurrently, but the token cost stays.
The lesson is not “always batch.” It is “batch the questions that can be answered from the same state without knowing each other's answers.”
Dependencies belong in code or a second request
A dependency looks like this: “If the ticket is a refund request, score whether the customer is within the refund window.” The second question needs the first answer, but Jev will answer both against the original state. You have three options.
First, send both questions in one request and let code decide which answer matters. This is speculative fan-out. It works when the second question is cheap and useful on several branches, and when you can tolerate paying for it on requests that never use it.
Second, split the request. Send the routing Choice first, branch in code, then send the follow-up against a state that now includes the routing result. This is the hierarchical classification and SDE cascade family: use a cheap decision to narrow the next decision, and only pay for the branch you take.
Third, remove the dependency. If the follow-up question can be rephrased as an independent judgment (“is this refund request inside the policy window?”), ask it directly and use the answer alongside the routing answer. Atomic questions are easier to evaluate and cheaper to debug.
The wrong option is the one that looks like a conversation: ask a question, wait, then ask the next question in the same request. Jev does not have a conversation history. The state guide is explicit that everything a question can see has to be reachable from the state.
Design the state for all branches
The one-request-many-questions article covers the batching mechanics. The design-side addition is that the state has to serve every branch you fan out to. If a refund branch needs the order date and the policy window, those fields have to be in the state before the first request, even if the routing answer later says the request is not about refunds.
That is a trade: a state that serves all branches is larger than a state tailored to one. The state design guide recommends including the material and the business facts the model cannot infer, and leaving out conclusions. The same rule applies here. Include the raw fields; do not include a pre-computed is_refund_eligible boolean that hands the model its answer.
For large taxonomies, the advanced structure guide recommends passing a subtree per option and trimming each branch to its direct children plus a sample of leaves. That keeps every candidate path available in one request without sending the whole taxonomy. The docs warn that untrimmed subtrees grow fast and inflate the input bill, which is the same trade-off in a different shape.
Measure the split, not only the answer
A two-request cascade has two budgets: the cost of the first request and the cost of the second, weighted by how often the branch fires. The composite-scoring pattern is a useful template for keeping the policy visible in code. The model returns components; the weights and the branch logic are yours.
Track three numbers for each split:
- Branch rate: how often the second request fires.
- Marginal value: how much the second answer changes the outcome compared with the first-request answer alone.
- Cost per resolved case: first-request cost plus branch rate times second-request cost.
If the marginal value is near zero, the second request is ceremony. If the branch rate is high, a single fan-out may be cheaper. If the branch rate is low and the second question is expensive, the split is doing its job.
The confidence notes add a useful guard: confidence is not accuracy. A high-confidence routing answer does not guarantee that the branch is the right one, so keep a review path for the cases where the routing answer and the follow-up disagree.
A short checklist
- Batch only questions that can be answered from the same state independently.
- Use speculative fan-out when several branches reuse the same extra judgments.
- Use a second request when the follow-up genuinely needs the first answer.
- Put the dependency in code when it can be expressed as a rule.
- Design the state for every branch the request might take.
- Measure branch rate, marginal value, and cost per resolved case.
- Version the question graph alongside the questions themselves.
The point of a decision model is to make one pass cheap. A chain of dependent requests gives that advantage back. Split when you must, batch when you can, and keep the dependency graph explicit.