Article
CommunityJev Tools & CLIs
Evaluate Agent Responses with Jev
Use Jev as a Braintrust judge scorer, then inspect selected answers, confidence, and probabilities alongside the scorer answer. Includes tracing for your own calls.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- Braintrust
- Last verified
- Sep 2026
Highlights
- Jev works as a Braintrust judge scorer via a TypeSafe API key provider, or native Jev once access is enabled for the account.
- Choice and Score confidence and per-choice probabilities are saved in the result metadata for later inspection.
- A worked example classified a draft as 'Incorrect, unsafe, or invents information' with 0.57 confidence and 0.68 probability.
- wrapTypeSafe(new TypeSafeClient()) records calls as typesafe.systemOne spans with input, criteria, answers, token usage, and duration.
- Suggested routing: accept confidence above 0.95, send 0.70-0.95 to a stronger LLM judge, and treat below 0.70 as inconclusive.
Quickstart
ts
const logger = initLogger({ projectName: "jev-evaluation" });
const client = wrapTypeSafe(new TypeSafeClient());
const response = await client.systemOne({
state: "I was charged twice. Please refund the duplicate charge.",
questions: {
category: choice("Which team should handle this request?", {
billing: "Payments and refunds",
technical: "Software problems",
other: "Other requests",
}),
},
});
await logger.flush();Watch out
Confidence values are not operating thresholds out of the box, and Braintrust says teams should compare them against observed accuracy on their own examples.
Questions about this entry
Short answers drawn from the linked project or page.
No, they are TypeSafe's own workflow evaluations, and the post repeats them as a reason to test rather than as proof. The one measured example here is a draft reply classified as Incorrect, unsafe, or invents information at 0.57 confidence and 0.68 probability.
There are two routes. Add TypeSafe as an organization provider with your own API key, or request access so native Jev appears among Braintrust's built-in models, which is not enabled by default. The judge takes text and either classifications or scored choices, in experiments and online scoring alike.
Not out of the box. The suggested bands are accept above 0.95, send 0.70 to 0.95 to a more capable LLM judge, and treat below 0.70 as inconclusive or route it to human review, and the post says these are not operating thresholds by default. Confidence summarises how concentrated the probabilities are, separately from the selected option's probability.
Yes. The selected choice, the model, confidence and per-choice probabilities are saved in result metadata, and wrapTypeSafe around a TypeSafeClient records your own calls as typesafe.systemOne spans with input, criteria, answers, token usage and duration, so you can inspect a questionable result and then refine the scoring criteria.
The customer request, the verified facts and the draft reply, then one instruction to pick the single label that best describes whether the response is ready to send. The worked example uses four labels, running from incorrect, unsafe or invented information through to an accurate answer that clearly explains the next step, so the failure modes are named rather than left to a free-text score.
Yes. The Jev judge integration accepts text and lets you configure either classifications or scored choices, and the post says the same criteria can be applied in experiments and in online scoring. Only the selected choice becomes the classification result or maps to the configured score, so a live trace and an experiment row are judged by one definition.
A typed answer rather than prose. Your code supplies the state and one or more constrained questions, Jev evaluates the independent questions in parallel, and Braintrust files the selected choice, the model, confidence and the per-choice probabilities in the result metadata. Nothing is written that you then have to parse back out of generated text.
More Jev resources like this
CLI and Python library that runs declarative YAML judgment tasks - classify, route, score, verify - natively on Jev Choice/Score/Noul, on any OpenAI-compatible backend, or on local keyword rules, with confidence gates, fallbacks and per-decision cost records.
Jev Tools & CLIs#community#yaml#classification
Community
lexingtonhibiki/judgekitPython
lexingtonhibiki/judgekitPythonCLI that collapses Loki log batches and asks Jev six typed questions in one call, then maps the answers in code to suppress, watch, review, notify, page or an auto-remediation candidate.
Jev Tools & CLIs#community#python#cli
Community
jyatesdotdev/jev-logtriagePython
jyatesdotdev/jev-logtriagePythonCloudflare Workers AI's model page for typesafe/jev: run the structured evaluation model through an env.AI.run binding or the REST /ai/run endpoint, with typed Noul, Choice, and Score questions and worked examples.
Jev Tools & CLIsDocs#community#cloudflare#integration
Community
@ai-sdk/typesafe-ai adds an evaluation model to the Vercel AI SDK: one experimental_evaluate call sends a state with choice, score, and boolean questions and returns typed answers, probabilities, confidence, and usage.
Jev Tools & CLIsDocs#community#vercel#integration
Community
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptLatitude is an open-source observability platform for AI agents: one line of telemetry captures traces, failing traces group into tracked signals, and fixes are dispatched to coding agents and replayed against real traces.
Jev Tools & CLIs#community#catalog-import#logicrw
Community
latitude-dev/latitude-llmTypeScript
latitude-dev/latitude-llmTypeScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.