Post
CommunityJev Benchmarks & Evaluations
An Independent Evaluation of TypeSafe's Jev
Vals AI's independent benchmarks of Jev: it ties frontier models on 400 claim-verification items drawn from SEC filings at about 1/500th of GPT-6 Astra's cost, but finishes last of twelve on a preregistered 12-subtask LegalBench slice.
- Format
- Post
- Reading time
- 2 min
- Published by
- Community
- Author
- Vals AI
- Last verified
- Oct 2026
Highlights
- On claim verification all twelve systems scored 0.953–0.995; Jev scored 0.975 at $0.02 per 1,000 cases, tying GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna.
- On LegalBench, Jev finished last of twelve on raw and class-balanced accuracy, and every other system's lead was statistically reliable.
- Jev answers in about a tenth of a second whether asked one judgment or thirty-two; at 32, median latency was 194x lower than GPT-6 Astra's.
- With a 1% error budget, six of twelve systems exceeded the limit on held-out items; Jev automated 95% of cases at 1.6% error.
- Input costs $0.042 per million tokens and output is free; the 400 items came from the latest 10-Ks of 48 large US companies.
Watch out
Grounded in the truncated Vals AI blog post read on 2026-10-15; the excerpt ends inside the McDonald's example results table. Vals built both benchmarks because its proprietary suite evaluates written answers, and every item's fixed answer was audited by a human before any system saw it.
Questions about this entry
Short answers drawn from the linked project or page.
Jev, TypeSafe's decision model, against frontier models on two purpose-built benchmarks: 400 claim-verification items made from SEC filings, and a preregistered 12-subtask slice of LegalBench. Twelve systems ran on both benchmarks.
Its proprietary benchmarks evaluate written answers, and Jev does not generate text: it takes a state plus questions and returns a probability distribution over allowed answers. So Vals built two benchmarks that fit a model with no text output.
It scored 0.975 at $0.02 per 1,000 cases and tied GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna. All twelve systems scored between 0.953 and 0.995, and the post calls Jev the cheapest model by far.
It finished last of twelve on raw accuracy and on class-balanced accuracy. On raw accuracy, every other system's lead over Jev is statistically reliable.
It answers in about a tenth of a second whether one judgment or thirty-two are requested. At thirty-two, its median latency was 194x lower than GPT-6 Astra's and 241x lower than Gemini 4 Argon's.
Six of twelve systems exceeded the limit on held-out items, and Jev was one of them: it automated 95% of cases at 1.6% error. The claim-verification cost per 1,000 cases was $0.02.
Input costs $0.042 per million tokens with output free, about 1/500th of GPT-6 Astra's cost on the claim-verification board. The evaluation was written by Connor Frank at Vals AI and published 10/06/2026.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Reddit: Localization QA with a decision model: meaning, terminology, and placeholders as separate checks.
Hacker News: Pick Verify: a single decision at the warehouse pick station catches confusable picks.
Show HN: Verifying robot picks with a decision modelRead the thread on Hacker NewsMore Jev resources like this
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
patchy631/jev-as-judgePython
patchy631/jev-as-judgePythonProphetLab's Texas Hold'em benchmark and playground for decision models: separate cash and sit-and-go leaderboards, hand replays, a Jev-backed advisor, and bring-your-own-agent tables, all with virtual chips.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
prophetlab/jevpokerbenchPython
prophetlab/jevpokerbenchPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonA minimal coding agent that filters every tool result through Jev before the main model reads it, plus the A/B harness that measured pass@1 and cost. The final TB4 follow-up closed the experiment without an end-to-end win.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
astro-han/jev-harnessPython
astro-han/jev-harnessPythonA benchmark for how efficiently decision models plan across multi-step problems: models that explore fewer unnecessary intermediate states score higher, and a perfect score means beelining along a shortest path. Seven domains, seeded trials and classic baselines.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
kyle-pena-nlp/beelinebenchPython
kyle-pena-nlp/beelinebenchPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.