Can I reproduce the results?

Yes, with a Vercel AI Gateway key in .env.local. npm run measure runs 60 cases x 3 rounds; npm run score writes summary.json, and npm run demo replays 12 measured cases in three lanes.

How are the two sides actually called, and how fair is that?

src/jev.ts calls Jev's experimental_evaluate while src/baseline.ts calls the comparison LLM through generateObject for structured output, and every model goes through the same one Vercel AI Gateway key so all of them share a gateway hop. Jev and gpt-4o-mini are called serially, one warm-up call per model is discarded, and there are no retries.

What do I need, and what does a full run cost?

Node.js, tested on v26.5.0, and an AI Gateway key starting with vck_. On the free tier you hit 429s and the run stops partway, and Claude is not available on the free tier at all, so add credits first. The 60 x 3 run of Jev plus gpt-4o-mini cost about $0.03, the Claude run about $0.62.

How much should I trust the labels and the cases?

The README is explicit that all 60 cases are synthetic with no real customers, bookings or contact details, and that labels were assigned by a single annotator, with hard cases carrying comments explaining the labeling call. The 20 pilot cases and the 12 demo cases were fixed mechanically before anyone saw results.

What do the reporting commands produce?

npm run score writes results/summary.json and results/per-case.json with accuracy, latency p50 and p95, cost and an operational simulation, npm run charts writes three SVGs for the blog post, and raw responses land under results/raw/<runId>/, which is gitignored. The replay demo shows 12 measured cases in three lanes, each taking exactly its measured round-1 latency.

Is the live demo the same as the measurement?

No, and the README says so. npm run demo replays already-measured cases at recorded latencies, whereas npm run demo:live calls the APIs on the spot with all lanes in parallel, which differs from the serial measurement conditions. There is also no license file, so the code is all rights reserved by default.

CommunityJev Benchmarks & Evaluations

Jev Eval — Shogo-nfrealmusic

Third-party TypeScript check that compares Jev (typesafe-ai/jev) against openai/gpt-4o-mini and anthropic/claude-sonnet-4.5 under identical conditions, routing 60 synthetic booking inquiries in four languages for a photo-shoot service, all through one Vercel AI Gateway key.

See highlights

Open github.com

Stars: 0
Language: TypeScript
Published by: Community
Last verified: Oct 2026

Shogo-nfrealmusic/jev-eval

User repository on GitHub

View on GitHub

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

A third-party check of Jev (typesafe-ai/jev), TypeSafe AI's judgment-only model, against LLMs (openai/gpt-4o-mini and anthropic/claude-sonnet-4.5) under identical conditions. The task: routing booking inquiries sent to a photo-shoot service for international tourists in Japan (60 synthetic messages in 4 languages). Node.js (tested with v26.5.0) and npm. A Vercel AI Gateway API key (starts with vck). Jev and the comparison LLMs are all called through this one key.

Quoted from the project's README. Read the full README.

Tagscommunity benchmark comparison routing

Works forJev for Routing Jev for Evaluation

Used inJev for Software & DevTools

Languages: TypeScript
Platforms: Vercel AI Gateway
Capabilities: Open sourceRequires API key

Highlights

Compares Jev with GPT-4o-mini and Claude Sonnet 4.5 under identical conditions on the same task
60 synthetic multilingual booking-inquiry routing cases with gold labels and per-case predictions
Every model is called through one Vercel AI Gateway key, so comparisons share the same gateway hop
A replay demo shows 12 measured cases in three lanes at their measured round-1 latency
Main run (60 x 3 rounds, Jev plus GPT-4o-mini) cost about $0.03; the Claude run about $0.62

Watch out

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.

FacebookClinicnotes

YouTube: Jev explained for routing and judging, with a comparison against LLMs.

Hacker News: Vercel on using Jev for defined decisions and GPT-6 Astra when the job also has to write or use tools.

flashbrew4 pointsTypeSafe AI Jev vs. GPT-6 AstraRead the thread on Hacker News

Hacker News: ASTGL replay chose local Laya routing over hosted Jev on one Mac Studio setup.

Jmeg8r15 pointsDoes an open-weight decision model beat a hosted one? Jev vs. LayaRead the thread on Hacker News

Related terms

Glossary definitions related to this entry.

Intent routing

Other listings of this project

Forks and copies of this project in the directory. This entry is the canonical one.

jev-eval

A measurement study that scores Jev against gpt-5.4-mini and gpt-5.6-luna on four public classification sets, then against recorded outcomes on a few thousand real pipeline decisions. Every per-item answer and the scoring code are committed.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

onlyoneaman/jev-evalTypeScript

More Jev resources like this

jev-regress-bench

Reproducible benchmark asking which approved agent answers changed meaning after a config edit: one Jev Choice (same, fact_differs, action_differs, specificity_differs) caught all 19 real changes with 13 false alarms across 109 before/after pairs, versus 33 for a three-tier stack and 19 for an LLM judge.

Jev Benchmarks & Evaluations#community#benchmark#regression-testing

Community

redhatpanda/jev-regress-benchTypeScript

OpenWork

24k

An open Claude Cowork-style workspace whose eval testkit can call Jev as a typed verification judge on agent-produced artifacts.

Jev Benchmarks & Evaluations#community#typescript#agents

Community

different-ai/openworkTypeScript

LangWatch

4.9k

LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.

Jev Benchmarks & Evaluations#community#typescript#evaluation

Community

langwatch/langwatchTypeScript

Jev Synthetic Survey

Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.

Jev Benchmarks & Evaluations#community#benchmark#calibration

Community

jjd-lab/jev-synthetic-surveyPython

S1MB

System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.

Jev Benchmarks & Evaluations#community#benchmark#leaderboard

Community