Why use Jev instead of an LLM judge?

The documented production replays show roughly half a second and a fraction of a cent per call with no malformed outputs, plus lower repeat-to-repeat variance than prose judges.

How do I catch vague assertions?

Borderline probabilities near the threshold are marked as such, so a p near 0.5 across repeats signals the assertion wording needs tightening rather than a model flake.

What happens on network or malformed Jev errors?

They return as error results with failed status and never count as passes, and if the explanation judge also throws, the failing verdict stands with the error attached.

Do I have to use LangChain or the Vercel SDK?

No. Core scenarios and assertions are provider-neutral, with optional adapters for LangChain models, Vercel providers, autoevals scorers, and tau-bench style benchmarking.

How do I install zevals, and which parts are optional?

npm install @zevals/core. LangChain models need @langchain/core, @langchain/openai and @zevals/langchain; Vercel AI SDK providers need ai and @zevals/vercel; autoevals scorers need autoevals and @zevals/autoevals.

How does dynamicMessage differ from user simulation?

dynamicMessage computes the next user message from the transcript at execution time, deterministically and for free, while userSimulation calls an LLM per turn and needs an until criterion. Both are segment types, and an exhausted simulation records an explicit failed result rather than staying silent.

What happens to attachments and metadata on a message?

User messages can carry an opaque context object. Zevals passes it through untouched to Agent.invoke and never sends it to judges, and a SyntheticUser's context is preserved by userSimulation.

CommunityJev Agents & MCP

Zevals

TypeScript evaluation utilities that test full AI agent conversations as end-to-end scenarios with binary assertions, without imposing a test runner.

See highlights

Open github.com

Stars: 11
Language: TypeScript
License: MIT
Published by: Community
Author: opencx-labs
Last verified: Oct 2026

opencx-labs/zevals

Organization repository on GitHub

View on GitHub

Simple, practical AI agent testing in TypeScript

Forks: 3
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Simple, practical AI evaluations in TypeScript. Zevals provides utilities for testing AI agents. Unlike the few existing AI eval libraries and frameworks: Treats AI evals like end-to-end tests, with less focus on metrics and more focus on binary assertions. Designed to evaluate full conversations, not just single query/response pairs. Does not impose any testing framework or test runner. Support for evaluating entire scenarios, with optional user simulation.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification Jev for Evaluation Jev for Tool calling

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

Segments compose user messages, agent responses, and judgments into scenarios, including deterministic adaptive replies.
Assertion helpers cover follow-up behavior and tool calls, with scoping to the latest assistant turn when needed.
Synthetic users simulate multi-turn humans until any pass or fail criterion, with explicit failure on turn ceilings.
Message context passes attachments and metadata to the agent untouched and is never forwarded to judges.

Watch out

Jev judges return probabilities without prose, so explanations need a separate judge called only on failures. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

cmd-mod-jev-nudge

Command Code mod that asks Jev at every agent stop whether a nudge would help, continuing unfinished work while leaving waits for the user alone.

Jev Agents & MCP#community#jevbest#catalog-import

Community

CommandCodeAI/cmd-mod-jev-nudgeTypeScript

jev-mcp

An MCP server built around measuring questions rather than asking them: jev_ask prototypes wording, jev_eval sweeps thresholds against labeled data, jev_map applies the winner over a workload.

Jev Agents & MCP#community#jevbest#catalog-import

Community

BYK/jev-mcpTypeScript

duet-agent

Agent harness for work that outlives one chat, with resumable turns, routed state machines, and observational memory across sessions and processes.

Jev Agents & MCP#community#jevbest#catalog-import

Community

dzhng/duet-agentTypeScript

skilled-agent-harness_spec-driven-loops

Assistant framework that stores specs, memory, and specialist agents as plain markdown files in your repo for Claude Code, Codex, Opencode, and similar runtimes.

Jev Agents & MCP#community#jevbest#catalog-import

Community