Capacity planning for decision workloads looks unfamiliar because the bill is denominated in input tokens, not requests. That changes the arithmetic. A single large request can consume more capacity than a hundred small ones, and a batch that adds questions without adding requests can raise cost while leaving request-per-second headroom untouched.
The models page states two rate limits: 100,000 tokens per second and 80 requests per second. Whichever binds first is your real limit.
Two ceilings, not one
The request ceiling is easy to reason about: 80 requests per second is 4,800 per minute, or roughly 6.9 million per day if you could sustain it. The token ceiling is harder because it depends on what you send. The same page advertises 250,000 tokens per second of model throughput, but that is a processing rate, not an allowance. The account-level limit is the 100,000 tokens per second figure, and it applies to input tokens.
Which ceiling binds depends on state size. A one-line classification might use 300 input tokens, so 80 requests per second costs only 24,000 tokens per second: request-bound. A 20,000-token document costs 1.6 million tokens per second at the same request rate: token-bound by a factor of 16. The fan-out benchmark found about 261 input tokens of fixed overhead per request, so even tiny states have a floor.
Do the division per endpoint, not per account. A routing endpoint with small states and a document endpoint with large states have different bottlenecks, and averaging them hides the one that will page you.
Estimate tokens per decision
A useful estimate has three parts: the state, the question set, and the fixed overhead. Measure all three on a representative sample. Do not use the character count divided by four as a production estimate; tokenization varies by language and structure.
The parallel questions cookbook ran 13 questions over a 54,000-character document for $0.000497 in one call. The fan-out benchmark ran 2,976 requests and about 2 million input tokens in roughly 15 minutes for about $0.09, with median savings of 76% to 86% at eight questions depending on state size. Those are useful anchors for a backfill, not for your peak-hour estimate.
The models page prices input at $42 per billion tokens and output at zero. The 64k context covers 32k of state plus the longest question. If a request exceeds the context, it does not degrade gracefully; it fails. Capacity planning therefore includes a size check before the call, not just a retry after it.
Concurrency is a queue, not a thread count
Little's law is the only formula you need: concurrency equals throughput multiplied by latency. At 80 requests per second and a 0.4 s p50, you need about 32 requests in flight on average. At the p95 of 1,477 ms reported in the jev-oncall triage run, the same throughput needs about 118 in flight. If your client pool is sized for the median, the p95 will queue.
That is why a capacity plan written in requests per second is incomplete. The vendor limits requests, but your client limits connections, your gateway limits concurrent streams, and your database limits writes. The slowest of those becomes the real ceiling.
The jev-oncall numbers are self-reported from a 300-alert run, and its caveats say the routing rules were revised after seeing results. Treat them as a shape, not a guarantee: p50 around 418 ms, p95 around 1,477 ms, and a tail that is more than three times the median.
Bursts need backoff and idempotency
A 429 is a capacity signal, not an error to retry immediately. The API reference lists 429 and 529 as the retryable responses and says to use exponential backoff; 401 and 422 are caller errors. The Python SDK defaults to two retries, 0.5 s initial backoff doubling to 5 s, jitter 0.25, and a 30-second total retry budget.
Those defaults are reasonable for a single interactive call and dangerous for a burst. If 1,000 workers all retry at once, the jitter helps but the herd is still a herd. Add a queue in front of the model, cap in-flight requests, and shed load before the vendor does it for you.
Idempotency is the other half. A retry after a timeout may have succeeded on the server. If the operation has side effects, the retry needs a key that the downstream system can deduplicate. For pure decisions, the risk is lower, but the cost is not: a retry storm is billed like any other request.
Fan-out changes request count
Batching is a capacity tool as well as a cost tool. Asking 13 questions in one request instead of 13 requests reduces request-per-second pressure by 13x, even though the token count is nearly the same. The speculative fan-out pattern makes this explicit: every question is evaluated in parallel, and adding questions usually has little effect on response time.
The catch is that batching increases the size of each request. If you are token-bound, batching does not help; it concentrates the same tokens into fewer, larger requests. If you are request-bound, batching is the cheapest capacity you can buy.
The fan-out benchmark also found that 1,000 questions succeeded in one call, with context limits bracketing 32,768 content tokens and 65,536 total. That is a wide envelope, but it is not infinite. A backfill that packs a whole corpus into one request will hit the context wall before it hits the rate limit.
Separate online and backfill capacity
Online traffic and backfills compete for the same account limit. A nightly reprocessing job that saturates the token ceiling will starve the interactive path, and the resulting p99 spike will look like a model regression.
Use separate keys or projects where the vendor supports it, and schedule backfills outside peak hours. The enterprise page says higher rate limits are available for teams and high volume, with a sales contact for throughput, data requirements, and VPC questions. That is a negotiation, not a self-serve setting, so plan the lead time.
For offline work, a CPU model can be a useful pressure valve. S1 LLM Auto-Router reports about 14 ms p50 and 47 ms p99 on four pinned EPYC cores, and 26 ms on four Xeon cores, for seven routing questions in one ONNX pass. The caveats are significant: the numbers are self-reported on the author's own held-out set, the blind set is only 100 requests with roughly plus or minus 8 points of noise, and the API price is a launch price. It is a different model, not a drop-in substitute for Jev quality.
Plan with p95, not average
A capacity plan should state the traffic shape, the token distribution, the p50 and p95 latency, the retry policy, and the headroom. Then it should be tested at the p95, not at the average.
The workflow evals report 0.4 s per case at $0.0004 across four workflows, which is a useful planning prior for a decision-shaped workload. The independent audit found a median round trip around 310 ms on six production-shaped calls and estimates Jev is roughly 76x cheaper and 25x faster than one frontier comparator at similar accuracy. Those are not capacity guarantees, but they are better anchors than a homepage demo.
The confidence docs are a reminder that a threshold change can alter the traffic mix: routing more cases to a second request or a human reduces model load but increases queue load. Capacity planning is not just model capacity. It is the capacity of the whole decision path.
The cost of a decision explains the input-token economics; retries and failure modes covers the backoff details; hosted versus self-hosted TCO covers the case where the rate limit itself is the reason to leave.
