A latency budget is not one number. It is a percentile, a traffic shape, and a boundary between what you control and what the vendor controls. Teams usually write the average they saw in a demo, then discover that the decision call sits behind a queue at the exact moment a user is waiting.
A percentile is the contract
The p50 tells you what a typical request costs. The p95 and p99 tell you what your users will remember. A 300-alert triage run in jev-oncall reports p50 418 ms and p95 1,477 ms: a tail more than three times the median, which is normal once retries and larger states enter the mix. The same repository is candid that its routing rules were revised after seeing results, so treat the number as an operational observation rather than a benchmark.
An independent playground operator measured six production-shaped calls at $0.000023 to $0.000038 each, with a median round trip around 310 ms, in its audit of TypeSafe's published numbers. TypeSafe's own homepage demo, by contrast, shows 0.114 s for one short, dense query, and the audit notes that the demo is chosen to flatter the model. Neither figure is a p99. If your product promise is a p99, you have to produce one on your own traffic.
The models page gives the vendor's throughput ceiling: 250,000 tokens per second and a 64k context covering 32k of state plus the longest question. That is capacity, not latency. It tells you how fast input can be consumed once the request is admitted; it says nothing about how long the request waits to be admitted.
Where the milliseconds go
A decision call has four cost centers: assembling the state, tokenizing and processing it, evaluating every question against it, and the network round trip. The model does the middle two. You do the first and often the last.
State size dominates the first two. A long ticket history re-sent for one classification is latency you chose. The fixed overhead is smaller than most people assume (the fan-out benchmark measured about 261 input tokens of overhead on every request), but the variable part grows with everything you paste in.
Questions are mostly parallel. The API reference describes one request with one state and a map of typed questions, and the answers come back keyed by the ids you supplied. Adding a question does not add a round trip. That is why the parallel questions cookbook could run 13 questions over a 54,000-character document in 0.27 s, against 2.71 s for 13 sequential calls.
The tail is rate limits and retries
The models page states two rate limits: 100,000 tokens per second and 80 requests per second. A request over either one returns 429. The API also uses 529 for capacity. Both are retryable, and the reference says to retry them with exponential backoff; 401 and 422 are not retryable because they are caller errors.
Retries are where a clean p50 turns into an ugly p99. The Python SDK defaults to two retries, 0.5 s initial backoff doubling to 5 s, jitter 0.25, and a 30-second total retry budget per call. That budget is sensible, but it means one unlucky request can consume 30 seconds of your latency allowance before it fails. If your p99 target is 2 seconds, a retry policy with a 30-second ceiling is not aligned with it.
The practical move is to separate interactive calls from batch calls. Interactive calls should have a short timeout and a deterministic fallback: a cached answer, a conservative default, or a human. Batch calls can afford the full retry budget.
Edge placement moves one hop
Running your handler at the edge puts the code close to the user. It does not put the model close to the user. If the decision endpoint is centralized, the edge shortens the client-to-handler hop and leaves the handler-to-model hop unchanged. For a 300 ms call, that may be 20 to 50 ms of real improvement; for a 1.5 s tail, it is noise.
Gateways add a hop of their own but can remove others. Vercel's AI Gateway evaluation path sends the same state and questions to /v1/evaluate, supports mixed question types in one round trip, and can constrain the request to one provider with only: ['typesafe-ai'] plus zeroDataRetention. OpenRouter's Jev listing is another route, priced at $0.042 per million input tokens with output free; it is useful as a failover path, not as a way to make compute local.
The residency question and the latency question are different. An edge function in Frankfurt does not make a US-hosted model process data in Frankfurt. If you need both, you need a regional data plane or a self-hosted model, not just an edge runtime. The TypeScript SDK at the edge covers the runtime constraints; the placement decision is a separate architecture choice.
Batching changes the shape of the budget
Sending five questions in one request is usually faster than sending five requests, but the reason is not that the model evaluates them faster in wall-clock terms. It is that you pay one network round trip, one state upload, and one admission decision. The cookbook's 10x speedup assumes the 13 single-question calls run sequentially; fired concurrently, the latency gap shrinks while the token cost stays.
Speculative fan-out takes advantage of the same property: ask everything the branch might need, then ignore what you do not use. The caveat is that every speculative question adds input tokens on every call. Latency stays roughly flat; cost does not.
This is also why a confidence threshold and a latency threshold should not be the same knob. A low-confidence answer may be worth a second request with more context; a high-confidence answer on a huge state may still be too slow for the interactive path. The confidence docs describe the first decision. Your p99 budget describes the second.
Measure the budget you actually have
Instrument four timestamps: request start, request sent, response received, and decision applied. Split them by endpoint, model version, state size bucket, question count, and retry count. Then run a load test at the traffic shape you expect, not at one request per second.
The measuring Jev piece makes the broader point that question design dominates quality; it also dominates latency, because a vague question invites a longer state and more second-guessing. The cost of a decision is the companion: the same design choices that shrink input tokens usually shrink the tail.
Research on compact inference is worth watching but not yet a procurement decision. FastJEV reports reducing candidate depth by 43.75% to 45.83% while retaining 93.66% to 97.52% of task scores, but the results are self-reported across six evaluation sets, no code is released, and the abstract notes that candidate prefix sharing can reduce computation while increasing latency. That is exactly the trade you should measure rather than assume.
Write the budget as p50, p95, and p99 for each route. Pin the model version. Count retries as part of the latency, not as an exception. Then decide whether the edge hop you are buying is worth the operational surface it adds.
