Why is this cheaper than a normal LLM-as-judge setup?

Input, output, and expected travel once as shared state with every rubric as a question in the same call, so cost scales with cases rather than cases times rubrics on a much cheaper per-token rate.

Do all rubric types reduce to pass or fail booleans?

No: boolean gives a probability of true, score gives a fractional value on the ordered scale, and choice gives the chosen key, each with thresholds in its own units.

How do I wire a support-agent suite into CI per pull request?

Define the suite with typed rubrics, thresholds, and cases with precomputed outputs or generate functions, then call runEval with concurrency; the CLI uses the success flag for its exit code.

How do I catch small regressions without flaky failures?

Compare saved and current runs with a tolerance that absorbs run-to-run noise, keeping fractional means unrounded so small regressions stay visible instead of rounding away.

How does authentication work in jev-evals?

It does not implement auth itself. It calls the ai SDK's experimental_evaluate, which resolves AI_GATEWAY_API_KEY first, then a VERCEL_OIDC_TOKEN from a linked Vercel project; without either, runEval throws AuthError before any calls.

What are the default concurrency and model for a run?

concurrency defaults to 5 cases in parallel, and the model defaults to typesafe-ai/jev. maxQuestionsPerCall, pricing, maxRetries, and an abort signal can be set per run.

How does compareRuns handle noise and missing rubrics?

It compares aggregate means rubric by rubric with a tolerance of 0.05 in the rubric's own scale, absorbing run-to-run noise. A rubric present on only one side is reported as new or removed rather than silently dropped.

CommunityJev Benchmarks & Evaluations

Jev Evals

Rubric-based eval harness that judges LLM and agent outputs with Jev through the Vercel AI Gateway, answering all rubrics for a case in one round trip.

See highlights

Open github.com

Stars: 3
Language: TypeScript
Published by: Community
Author: NicolasMontone
Last verified: Oct 2026

NicolasMontone/jev-evals

User repository on GitHub

View on GitHub

Rubric-based eval harness cheap enough to run on every PR, powered by typesafe-ai/jev

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

A rubric-based eval harness for LLM/agent outputs, backed by the typesafe-ai/jev evaluation model via the Vercel AI Gateway. LLM-as-judge normally costs about as much as the generation it's judging — one prompt in, one verdict out, per rubric. That symmetry is exactly why nobody runs evals on every commit: doubling your generation cost to check your generation quality doesn't survive contact with a budget. Jev breaks that symmetry two ways:

Quoted from the project's README. Read the full README.

Tagscommunity github

Works forJev for Evaluation

Used inJev for Software & DevTools

Languages: TypeScript

Highlights

Sends one shared state per case with boolean, score, and choice rubrics answered together in a single call.
Validates suites up front with typed definitions, per-case generation, thresholds, and unique case ids.
Reports per-case and per-rubric aggregates with token usage, real cost, timing, and run comparison.
Preserves fractional scores end to end so small quality shifts survive aggregation and diffing.

Watch out

Cost arithmetic in the repo is illustrative for its example suite sizes, so estimate from real usage data for your own rubric counts. Copy reviewed against the repository README on 2026-10-04, not from running it.