PatternsCanary deployment

Canary rollout

Acting on the new model for a small slice of traffic and watching the outcome metric, with a one-flag rollback to the incumbent.

Open docs.typesafe.ai All 164 terms

Category: Patterns
Also known as: Canary deployment
Related terms: 5
Directory entries: 39
Docs: docs.typesafe.ai
Added: 2026-10-12

Definition

A canary promotes the candidate from observation to action on a fraction of traffic — one percent, one queue, one region — and watches the metric that motivated the change. It is where shadow-mode confidence becomes real consequences on a bounded audience.

The decision-model-specific requirement is a pinned version: because a model alias can move under you, the canary and its control must each be fixed, or the comparison drifts mid-test. Rollback has to be a single switch, or it will not fire when it should.

Tagspatternsevaluation

Related terms

Definitions that connect to this one.

Shadow modeRunning a new model beside the incumbent and logging its answers without acting on them, to learn its distribution before trusting it.Patterns Fail closedWhen the model errors, times out, or lacks evidence, the system escalates or blocks instead of allowing the action.Ecosystem Confidence gateA threshold on confidence that decides whether code acts, confirms first, or escalates to a person or a bigger model.Answers & Confidence RepeatabilityHow stable answers are across repeated evaluations, measured separately from accuracy and reported as probability standard deviation.Evaluation & Training Model aliasA name like jev-latest that resolves to a versioned model ID and can move when a new release ships.API & Models

From the Jev directory

Workflow evals

Published evaluations of four automation workflows, security incidents, agent trace observability, invoice processing, and customer service, comparing Jev and frontier LLMs as structured workflows versus single prompts.

Jev Practices & PatternsDocs#official#benchmarks#evaluation

Official

Jev for QA

Katalon's guide to Jev in software testing: where bounded judgments help QA, why deterministic assertions stay first, the documented weak spots that hit test checks, and the caveat that a constrained answer can still be wrong.

Jev Guides & ArticlesArticle#community#testing#qa

Community

Python SDK Responses

The Python response models: SystemOneResponse groups answers by id, and records the model name, token usage, and request id.

Jev Repos & SDKsDocs#official#docs#sdk

Official

Companion

A 3D robot that understands without generating text: one TypeSafe call per turn with nine typed questions, then code decides whether to act, ask, or shrug.

Jev CookbooksPlayground#community#playground#embodied-agent

Community

jevcal

CLI that fits a per-question confidence threshold to a target accuracy on your own labeled data, verifies it on a held-out split, estimates how much traffic still needs an LLM, and re-checks locked thresholds in CI. It publishes no Jev results of its own.

Jev Practices & Patterns#community#python#calibration

Community

abhixhek/jevcalPython

typesafe-sdk (Ruby client)

A Ruby client for TypeSafe's System One API that returns typed Noul, Choice, and Score answers, validates obvious input mistakes before spending a request, and pools HTTP connections across threads.

Jev Repos & SDKs#community#ruby#sdk

Community

joshmn/typesafe-sdkRuby

33 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai