Article
CommunityJev Guides & Articles
Braintrust docs: Trace Jev and use it as an LLM judge
Braintrust's provider page covers two ways to use TypeSafe Jev: instrumenting System One calls as decision spans, and running Jev as an LLM judge that turns typed choices into scores or labels.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- Braintrust
- Last verified
- Oct 2026
Used inJev for Software & DevTools
Highlights
- Requires Braintrust v3.34.0+ and @typesafe-ai/sdk v0.6.0 or later in the v0.x line, plus a TypeSafe account and API key.
- Calls appear as typesafe.systemOne spans labeled Decision spans, recording state, questions, answers, token usage, duration and errors.
- Jev as judge turns a selected choice into a numeric score, pass to 1 and fail to 0, or into a classifying label such as billing.
- Built-in Jev draws on organization model credits and is not available for BYOC or self-hosted deployments, which need your own TypeSafe key.
- The docs recommend a general-purpose LLM for broad criteria like tone or helpfulness, where the useful output is a judgment, not a selection.
Watch out
Copy reviewed against the published documentation page on 2026-10-17. It is vendor-authored documentation for Braintrust's own integration, so setup details and behavior descriptions reflect Braintrust's and TypeSafe's product documentation rather than independent testing.
Questions about this entry
Short answers drawn from the linked project or page.
Trace Jev calls from your application to inspect their inputs, answers and probabilities in decision spans, and use Jev as an LLM judge that checks AI output against your criteria and turns its decision into a score or label in experiments or online scoring.
Braintrust v3.34.0+ and @typesafe-ai/sdk v0.6.0 or later within v0.x, plus a TypeSafe account and API key. Set BRAINTRUST_API_KEY and TYPESAFE_API_KEY in your shell; for the EU data plane also set BRAINTRUST_API_URL to https://api-eu.braintrust.dev.
Use Braintrust's import hook: node --import braintrust/hook.mjs trace-typesafe-auto.js. The docs note the auto-instrumentation example is plain JavaScript so node can run it directly; in TypeScript projects you compile first, or use wrapTypeSafe() for manual instrumentation.
Input state and questions with their identifiers, instructions and criteria; structured answers including choices, scores, Noul values, confidence and probabilities; model and provider metadata; token usage and request duration; and errors raised by the call.
No. TypeSafe confidence is a value between 0 and 1 that summarizes how concentrated the probability distribution is, separate from the selected option's probability and from an evaluation score, and the docs warn high confidence does not guarantee a correct decision.
Jev suits frequent, focused checks with a clear answer, such as whether a support reply follows your refund policy. For broad or nuanced criteria like tone or helpfulness, where the useful output is a judgment rather than a selection, the docs say a general-purpose LLM is the better judge.
It judges text, so media and tool message content are rejected; to judge a trace that contains tool calls, you pass the relevant context as text. Selecting Jev also hides the model parameter and tool controls, because it takes no sampling parameters and selects without a tool schema.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: Simon Willison's overview of Noul, Choice, and Score, and where a decision model fits.
benwerd53 pointsJev introduces a new shape of LLMRead the thread on Hacker NewsHacker News: swyx posts the Latent Space interview with Diogo Almeida.
swyx4 pointsJev: System One Models for Prod, Not God – With Diogo Almeida, CEO, TypeSafe AIRead the thread on Hacker NewsX: Diogo Almeida in five minutes
X: LLMs vs. Jev, clearly explained
More Jev resources like this
Langfuse's guide to decision-model evaluators runs OpenAI's gpt-6-luna or TypeSafe Jev over observations, where each Choice, Score or Yes/no question becomes one Langfuse score.
Jev Guides & ArticlesArticle#community#press#explainer
Community
The desk's index of every Jev measurement it has published, keeping each figure's original caveat: JevBench boards, Vercel gateway reach, Celeris and Artificial Analysis results, and the desk's own Jev Day probes which it says it did not rerun.
Jev Guides & ArticlesArticle#community#press#calibration
Community
Ian Maurer's ThinkThen analysis argues Jev's faster and cheaper are measurable today while its better is a bet on RLCD: unlike RLHF, which pleases humans, and RLVR, which optimizes benchmarks, Jev is trained to be reliable for programmatic use, in Diogo Almeida's own framing.
Jev Guides & ArticlesArticle#community#press#explainer
Community
A podcast episode on Jev as a decision model rather than a chat model: hosts Luis Calderon and Carina discuss roughly 20x faster Coach testing, why that preliminary figure differs from TypeSafe's vendor benchmarks, and why type safety does not guarantee a correct decision.
Jev Guides & ArticlesArticle#community#press#podcast
Community
A BIG CHANGE analysis of the Diogo Almeida Latent Space interview: Jev's most consequential proposition is the unit of automation, a bounded judgment inside an existing process, and the launch claim that Jev can't hallucinate needs a narrow reading because schema matching does not establish truth.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Langfuse's notebook integration traces TypeSafe Jev System One calls through the OpenInference instrumentor, which patches the Python client in place and emits OpenTelemetry spans.
Jev Guides & ArticlesArticle#community#press#explainer
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.