Jev Benchmark

Jev-style medical QA implementation on Qwen3.5-4B that selects an answer from candidates in one forward pass instead of generating text. It reports 69.42% accuracy versus 67.41% for standard generation and a 10.37x speedup across 153,889 questions from nine benchmark sets.

Third-party TypeScript check that compares Jev (typesafe-ai/jev) against openai/gpt-4o-mini and anthropic/claude-sonnet-4.5 under identical conditions, routing 60 synthetic booking inquiries in four languages for a photo-shoot service, all through one Vercel AI Gateway key.

Pre-registered benchmark adding Jev 1.13 to Stanford MedHELM's MedHallu results: one Noul asks whether a proposed answer misrepresents its PubMed abstract. Jev scored 92.9% on 1,000 test items at a 204 ms median and USD 0.03 per 1,000 checks through OpenRouter.

The first Chinese-language calibration benchmark for Jev-class System One decision models: 378 items and 467 questions across voice routing, e-commerce service and content moderation, with accuracy, ECE, Brier, NLL, AUROC, option-order and zh-CN/zh-TW robustness metrics.

Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.

Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.

System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.

Small test harness and benchmark for Jev through OpenRouter's Decisions API. It ships a thin decide wrapper, 17 documentation-derived cases with 23 checks covering Noul, Choice and Score, and a script comparing Jev against GPT-5.6 Terra and Claude Opus 5.

An open benchmark runtime for document-grounded decision models: it scores Jev Noul, Choice and Score answers on pinned tasks and counts malformed probability distributions as misses so model comparisons stay reproducible.

A benchmark that steers decision models through 100 key-and-door grid puzzles with one Jev Choice per move, scored against an exact BFS solver; it found Jev picks an optimal move about 90% of the time yet finishes only 57 of 100 games.

Reproducible benchmark asking which approved agent answers changed meaning after a config edit: one Jev Choice (same, fact_differs, action_differs, specificity_differs) caught all 19 real changes with 13 false alarms across 109 before/after pairs, versus 33 for a three-tier stack and 19 for an LLM judge.

Grounded information extraction library: deterministic code proposes candidate spans with exact offsets and Jev answers one Choice per span plus sentence-level Nouls, so every extraction is a substring of the source. Ships a bilingual benchmark and paper.

Veris tests Jev as a semantic turn-detection gate for one Pipecat voice agent from its VAmoS Pro Bench, with labels complete, short, and long. Interruptions fell from 52% to 11% of caller turns across noise conditions, but task completion barely moved and median replies slowed from 2.1 s to 4.6 s.

Halv reports a paired SWE-rebench comparison where its workflow, an Astra coordinator, a Luna investigation stage, and JEV selecting worker models and reasoning effort, recorded $144.67 in model cost versus $337.50 for vanilla Codex across 42 runs, with both arms passing 25 verifier runs.

Vercel's Pranit reports benchmarking the fx auto-mode safety classifier with Jev: about 5 to 18 times faster and more accurate than GPT-5.6 Luna.

An internal benchmark report: Jev zero-shot came within about 5 points of recall of a fine-tuned model at matched precision, at roughly $70 a month.

Guillermo Rauch says the fx auto-mode safety reviewer, currently on GPT Luna, is up to 18x faster at p95 and more accurate with Jev, and that Jev is coming to Vercel AI Gateway.

Vercel CTO Malte Ubl reports that Jev beat an existing classifier eval previously run on Gemini 2.5 Flash Lite, saturating the eval on quality and running 6x faster.

The WebMCP benchmark author reports that Jev paired with the small Mercury 2.5 model solved 100% of tasks at roughly 112x lower model cost than GPT-6 Astra with computer use.