A decision model in production spends more of its life failing than you expect. Rate limits, transient 5xxs, timeouts, and a provider that is simply down for a minute all arrive eventually. The question is not whether to handle them but which way to fail — and "it depends" is the only honest answer, so it has to be a deliberate one.
Fail open or fail closed, per path
The glossary names the two postures. Fail closed means no answer, no action: the decision is deferred, the user waits, the queue holds. Fail open means proceed on a default: allow the action, treat it as not urgent, keep the pipeline moving.
Neither is universally right. A content-moderation gate that fails open lets everything through during an outage. The same gate failing closed blocks all traffic during an outage. Which outage hurts more is a business decision, and it has to be made per path, not globally.
The Katalon testing guide puts the test-loop version plainly: default to unknown, never to pass, so a model outage cannot silently turn into a green run.
Retries belong in the client, with backoff
The official SDKs already retry idempotent failures with exponential backoff, and the retries docs list which errors are retried and which are not. Let the client do it, and tune the budget to your latency tolerance rather than reimplementing a backoff loop.
An important distinction the exceptions reference draws: a rate limit or a timeout is retryable, but a malformed request or a bad key is not. Retrying a 400 just burns your rate-limit budget faster.
Bound the whole call, not just the retries
A retry budget without a deadline is a way to make a slow outage into an unbounded one. Cap the total time a decision may take, and when it is exceeded, take the fail-open or fail-closed path you already chose. The Python client accepts both a per-request timeout and a retry count; set them together so the arithmetic is visible.
client = Typesafe(
max_retries=2,
timeout=2.0, # seconds, covers the whole call including retries
)
Make the default explicit
The dangerous failure is an implicit one. If your handler catches every exception and returns False, you have silently chosen fail open without saying so, and nobody reviewing the code will see the policy. Write the fallback as a named function that states the posture, and log every time it fires.
decision = escalate_or_default(state, default=False) # fail open: do not escalate
Watch the error rate, not just the latency
A model that is up but timing out under load looks healthy on a latency dashboard until the timeouts dominate. Track retries and fallback invocations as first-class metrics. If the fallback path is running more than a fraction of the time, the effective policy of your system is the fallback, not the model.
When a rival endpoint is the fallback
Because the decision API shape is portable — Cloudflare's Clef and the open servers answer the same typed questions — a fallback can be a second base URL rather than a hard default. That keeps the judgment in a model when one provider blips, at the cost of a second key and the calibration work of matching a second model to the same thresholds. The local serving article covers what stays the same when you switch the endpoint.
Decide the posture per action, put retries in the client, bound the total time, name the default, and measure how often it fires. Reliability here is a set of explicit choices, not a retry library.