Docs
CommunityJev Tools & CLIs
TypeSafe Provider
@ai-sdk/typesafe-ai adds an evaluation model to the Vercel AI SDK: one experimental_evaluate call sends a state with choice, score, and boolean questions and returns typed answers, probabilities, confidence, and usage.
- Format
- Docs
- Reading time
- 3 min
- Published by
- Community
- Last verified
- Sep 2026
Works forJev for Scoring
Used inJev for Software & DevTools
Highlights
- Install @ai-sdk/typesafe-ai, set TYPESAFE_AI_API_KEY, and evaluate with typeSafeAi.evaluationModel('jev-latest').
- Choice takes 1-255 options, Score 2-10 ordered levels, and the SDK's boolean maps to TypeSafe's Noul as P(true).
- Answers arrive rounded to two decimals; result.rounding reports the precision, so probability sums can miss exactly one.
- Confidence for Choice and Score sits under result.providerMetadata.typesafe.confidence[questionId], apart from probabilities.
- Failures raise APICallError; the SDK retries 429 and 529 up to maxRetries (default 2), with no provider-side retry loop.
Quickstart
ts
import { typeSafeAi } from '@ai-sdk/typesafe-ai';
import { experimental_evaluate } from 'ai';
const result = await experimental_evaluate({
model: typeSafeAi.evaluationModel('jev-latest'),
state: { message: 'I was charged twice. Please refund the duplicate.' },
questions: {
department: {
type: 'choice',
instructions: 'Which team should handle this?',
criteria: { billing: ['Charges', 'Refunds'], technical: ['Bugs'], other: null },
},
},
});
console.log(result.answers.department.choice);Watch out
Only the experimental evaluation API is supported - language, embedding, and image model factories throw NoSuchModelError - and the page is Vercel's, not TypeSafe's.
Questions about this entry
Short answers drawn from the linked project or page.
Evaluate only. The provider exposes the experimental evaluation API through experimental_evaluate with typeSafeAi.evaluationModel('jev-latest'), and the language, embedding and image model factories throw NoSuchModelError. There are no provider-specific options either: unknown entries under providerOptions.typesafe just produce unsupported-option warnings rather than doing anything, which is worth knowing before you build configuration around them.
No. TypeSafe returns scores and probabilities rounded to two decimal places and the SDK passes those values through unchanged, so a distribution can miss exactly one. result.rounding reports the precision so the SDK can account for rounding when checking sums and weighted means. Normalize deliberately if you need a clean distribution, rather than assuming the raw numbers already add up.
They are different statistics and mixing them up is the easy mistake. Confidence is a separate TypeSafe number available only for Choice and Score answers, under result.providerMetadata.typesafe.confidence[questionId]. A boolean answer is always P(true), never confidence in either outcome. The provider documentation is explicit that decision thresholds belong in application code, so nothing here picks a cutoff for you.
Choice takes 1 to 255 options, Score takes 2 to 10 ordered rubric levels, and boolean requires a model-estimated probability of true with optional true/false criteria. State, instructions and descriptions accept JSON objects and arrays in addition to strings, and descriptions may be null, which is how you express an open-ended option without inventing a description for it. All questions go in one request against the same state.
Install it with pnpm add @ai-sdk/typesafe-ai and set TYPESAFE_AI_API_KEY. createTypeSafeAi accepts apiKey, baseURL, headers and a custom fetch, and there is a default typeSafeAi instance if you only need the environment variable.
Yes. Set TYPESAFE_AI_BASE_URL, or pass an explicit baseURL to createTypeSafeAi, which takes precedence. The configured server receives the API key and decision state, so it must implement the TypeSafe POST /systemone route.
Provider failures raise APICallError, including validation and authentication errors. The core retries transient failures such as 429 and 529 according to maxRetries, which defaults to 2, and there is no provider-side retry loop. abortSignal, headers and custom fetch are forwarded.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Vercel ships the AI SDK provider for Jev
X: Jev lands on the Vercel AI Gateway
YouTube: Greg Isenberg and Ryan Vogel on the businesses Jev unlocks, and a path in through the Vercel AI Gateway.
More Jev resources like this
Vercel Labs' terminal CLI for the AI SDK: its `ai evaluate` command answers named Boolean, Choice, and Score questions, with Jev as the default evaluation model and one unchanged input shared across all questions.
Jev Tools & CLIs#community#typescript#cli
Community
vercel-labs/ai-cliTypeScript
vercel-labs/ai-cliTypeScriptCloudflare Workers AI's model page for typesafe/jev: run the structured evaluation model through an env.AI.run binding or the REST /ai/run endpoint, with typed Noul, Choice, and Score questions and worked examples.
Jev Tools & CLIsDocs#community#cloudflare#integration
Community
Use Jev as a Braintrust judge scorer, then inspect selected answers, confidence, and probabilities alongside the scorer answer. Includes tracing for your own calls.
Jev Tools & CLIsArticle#community#evaluation#observability
Community
AI Gateway's evaluation modality: call Jev as `typesafe-ai/jev` through the AI SDK's experimental_evaluate API, with structured answers instead of free-form text.
Jev Tools & CLIsDocs#community#vercel#integration
Community
An experimental Sutro CLI that evaluates CSV, Parquet, and JSONL data with Jev, asks you to label ambiguous and random audit rows, and uses GEPA to propose improved function definitions.
Jev Tools & CLIs#community#python#cli
Community
sutro-sh/jev-alignPython
sutro-sh/jev-alignPythonA CLI that scores code quality by asking Jev yes/no questions about each file, runs your existing test command, and turns uncovered paths into small actionable queries for agents.
Jev Tools & CLIs#community#rust#cli
Community
supercorp-ai/supercovRust
supercorp-ai/supercovRustFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.