PatternsShadow deployment

Shadow mode

Running a new model beside the incumbent and logging its answers without acting on them, to learn its distribution before trusting it.

Open docs.typesafe.ai All 164 terms

Category: Patterns
Also known as: Shadow deployment
Related terms: 5
Directory entries: 38
Docs: docs.typesafe.ai
Added: 2026-10-12

Definition

In shadow mode every request that reaches the incumbent also reaches the candidate, whose answers are recorded but never used. It collects real traffic distributions — where the models disagree, how confident the newcomer is on your states — with no user-visible risk.

Because output is free and the state is already assembled, shadowing a decision model is cheap, so a long shadow period is affordable. It is the stage where a threshold is chosen from data instead of guessed, before any user sees the new answer.

Tagspatternsevaluation

Related terms

Definitions that connect to this one.

Canary rolloutActing on the new model for a small slice of traffic and watching the outcome metric, with a one-flag rollback to the incumbent.Patterns Confidence gateA threshold on confidence that decides whether code acts, confirms first, or escalates to a person or a bigger model.Answers & Confidence Distribution shiftWhen the traffic a model sees at run time stops looking like the data it was calibrated on, so its probabilities no longer mean what they did.Evaluation & Training ThresholdThe value your code compares against a score, a Noul probability, or a confidence to turn an answer into an action.Answers & Confidence Fail openWhen Jev is down or slow, the gate allows the action, including rules that would otherwise block it.Ecosystem

From the Jev directory

Confidence

How TypeSafe reports certainty, how it differs from probability, and how to use it architecturally to gate and route decisions.

Jev Practices & PatternsDocs#official#confidence#routing

Official

Jev for QA

Katalon's guide to Jev in software testing: where bounded judgments help QA, why deterministic assertions stay first, the documented weak spots that hit test checks, and the caveat that a constrained answer can still be wrong.

Jev Guides & ArticlesArticle#community#testing#qa

Community

Python SDK Responses

The Python response models: SystemOneResponse groups answers by id, and records the model name, token usage, and request id.

Jev Repos & SDKsDocs#official#docs#sdk

Official

Companion

A 3D robot that understands without generating text: one TypeSafe call per turn with nine typed questions, then code decides whether to act, ask, or shrug.

Jev CookbooksPlayground#community#playground#embodied-agent

Community

jevcal

CLI that fits a per-question confidence threshold to a target accuracy on your own labeled data, verifies it on a held-out split, estimates how much traffic still needs an LLM, and re-checks locked thresholds in CI. It publishes no Jev results of its own.

Jev Practices & Patterns#community#python#calibration

Community

abhixhek/jevcalPython

typesafe-sdk (Ruby client)

A Ruby client for TypeSafe's System One API that returns typed Noul, Choice, and Score answers, validates obvious input mistakes before spending a request, and pools HTTP connections across threads.

Jev Repos & SDKs#community#ruby#sdk

Community

joshmn/typesafe-sdkRuby

32 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai