Docs
CommunityJev Tools & CLIs
Vercel AI Gateway: Evaluation Models
AI Gateway's evaluation modality: call Jev as `typesafe-ai/jev` through the AI SDK's experimental_evaluate API, with structured answers instead of free-form text.
- Format
- Docs
- Reading time
- 3 min
- Published by
- Community
- Last verified
- Sep 2026
Works forJev for Classification
Used inJev for Software & DevTools
Highlights
- experimental_evaluate (aliased from the ai package) needs AI SDK 7 or later; it is not supported through OpenAI-compatible endpoints.
- Boolean answers return { type: 'boolean', probability }; Choice returns choice plus probabilities for every option.
- Questions of different types share one state and are answered in one round trip, so adding questions doesn't need extra calls.
- POST /v1/evaluate at ai-gateway.vercel.sh takes the same model, state, and questions for non-AI-SDK callers.
- providerOptions.gateway accepts zeroDataRetention and only: ['typesafe-ai'] to constrain which provider serves the request.
Quickstart
ts
import { experimental_evaluate as evaluate } from 'ai';
const result = await evaluate({
model: 'typesafe-ai/jev',
state: 'The support agent issued a full refund.',
questions: {
refunded: { type: 'boolean', instructions: 'Was a refund issued?' },
},
});
console.log(result.answers.refunded.probability);Watch out
Evaluation is not supported through AI Gateway's OpenAI-compatible, Anthropic-compatible, or Cohere-compatible endpoints, so those callers must use experimental_evaluate or the HTTP API.
Questions about this entry
Short answers drawn from the linked project or page.
Evaluation is not served through the OpenAI-compatible, Anthropic-compatible or Cohere-compatible endpoints. Through the AI SDK it needs experimental_evaluate from the ai package on AI SDK 7 or later; everyone else posts to /v1/evaluate on ai-gateway.vercel.sh with the same model, state and questions shape.
Yes. providerOptions.gateway accepts zeroDataRetention and only: ['typesafe-ai'] to constrain which provider serves the request. The page also notes that your use remains subject to TypeSafe AI's terms and privacy policies, so the gateway setting is not the whole answer for compliance.
No. A boolean question returns type boolean with a probability, a Choice returns the selected option plus a probability for every option, and several questions can be answered in parallel in one request against the same state. Optional criteria on a boolean define the true and false cases, such as exit code 0 versus any non-zero code.
The captured page does not say. It states only that several questions can be answered in parallel against the same state and never describes how state tokens are billed. Check the model page for typesafe-ai/jev and the Evaluation quickstart before you build volume pricing on the assumption.
Boolean returns a probability between 0 and 1 and accepts criteria that define the true and false cases. Choice returns the selected option plus a probability for every option, and Score rates the state against ordered descriptions. Each key in questions becomes a key in answers.
Yes. Decision models were previously called evaluation models, and experimental_evaluate and gateway.evaluationModel() still work as deprecated aliases with the same request and response shapes, so existing code keeps running unchanged.
Use the Decision filter on the AI Gateway Models page to list the supported decision models, and follow the Decision quickstart for setup. Decisions are reachable through the AI SDK, the HTTP API, the OpenAI-compatible Decisions API and the TypeSafe-compatible API.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Jev lands on the Vercel AI Gateway
X: Vercel ships the AI SDK provider for Jev
YouTube: Greg Isenberg and Ryan Vogel on the businesses Jev unlocks, and a path in through the Vercel AI Gateway.
More Jev resources like this
@ai-sdk/typesafe-ai adds an evaluation model to the Vercel AI SDK: one experimental_evaluate call sends a state with choice, score, and boolean questions and returns typed answers, probabilities, confidence, and usage.
Jev Tools & CLIsDocs#community#vercel#integration
Community
Vercel Labs' terminal CLI for the AI SDK: its `ai evaluate` command answers named Boolean, Choice, and Score questions, with Jev as the default evaluation model and one unchanged input shared across all questions.
Jev Tools & CLIs#community#typescript#cli
Community
vercel-labs/ai-cliTypeScript
vercel-labs/ai-cliTypeScriptCLI and Python library that runs declarative YAML judgment tasks - classify, route, score, verify - natively on Jev Choice/Score/Noul, on any OpenAI-compatible backend, or on local keyword rules, with confidence gates, fallbacks and per-decision cost records.
Jev Tools & CLIs#community#yaml#classification
Community
lexingtonhibiki/judgekitPython
lexingtonhibiki/judgekitPythonA local screening workbench for a folder of CVs: it asks Jev a small fixed set of typed questions about each candidate, scores the answers with plain arithmetic you can read, and shows a sortable shortlist.
Jev Tools & CLIs#community#typescript#classification
Community
gtaras7/typesafe-jevTypeScript
gtaras7/typesafe-jevTypeScriptTypeSafe's announcement that Jev is available on the Vercel AI Gateway, linking the model page with the Jev API, pricing, and playground.
Jev Tools & CLIsPost#official#x#vercel
Official
Guillermo Rauch says the fx auto-mode safety reviewer, currently on GPT Luna, is up to 18x faster at p95 and more accurate with Jev, and that Jev is coming to Vercel AI Gateway.
Jev Tools & CLIsPost#community#x#vercel
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.