Is this benchmark run by TypeSafe?

No. It is Benchmark Heaven's own benchmark and is not affiliated with or endorsed by TypeSafe AI, whose Jev model is one of the systems measured — currently listed at 63.29 behind Imajev-4B, Plumb-4B, and decider-4b v2. The harness is MIT for the scoring code, while other artifacts keep their own licences.

Are the latency and cost columns measured, or estimated?

Mixed, and the repo labels it. Latency of self-hosted and demo endpoints is adjusted by x2, plus 0.15 s, to approximate production load — an assumption, not a measurement; raw numbers live in the results files. Cost is dollars per 1,000 decisions, not per 1,000 tokens, so a system with hundreds of tokens per question looks far more expensive than a per-token reading would suggest.

Can a small model just game the option order?

Yes, and it is documented. One entrant answered with its own option order (A. yes, B. no) at 72% on answer-judging items; reverse the order to (A. no, B. yes) and the same model scored 21%. Small models are very sensitive to option order, so a rank is not a portable claim about your own prompts.

Is the held-out half still held out?

The hard tier is 220 decisions, 111 public and 109 held out, written by two named frontier models, cross-reviewed, frozen and hashed before any system ran, and only aggregate sealed statistics are published. The caveats are real though: the public half can be trained on or selected against, and rows flagged API are operator endpoints that received item text without answer keys.

What is the difference between the Capability and Composite scores?

Capability Score is the arithmetic mean of Intelligence and Calibration within official caps and headlines the open-weights board, while Composite Score is a weighted harmonic mean of Intelligence, Calibration, Speed and Cost and leads the hosted API board.

How is this JevBench different from the one at jevbench.github.io?

This JevBench is maintained by Benchmark Heaven at benchmarkheaven.com; the metamorphic coherence benchmark at jevbench.github.io is a separate project whose consistency scores measure a different property.

What happens to a missing measurement on the JevBench board?

Missing measurements are null, never zero, and the repo warns against comparing scores across method versions as if they were the same experiment, since historical releases keep their original scoring definitions.

Jev directory

CommunityJev Practices & Patterns

JevBench — fstandhartinger

Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.

See highlights Open github.com

Stars: 282
Language: Python
License: MIT
Reading time: 3 min
Published by: Community
Author: fstandhartinger
Last verified: Sep 2026

fstandhartinger/jevbench

User repository on GitHub

View on GitHub

JevBench by Benchmark Heaven: benchmark and leaderboard for AI decision models, measuring typed decision accuracy, calibration, latency and cost.

Forks: 27
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

Jev is TypeSafe AI's decision model; JevBench also evaluates Jev-compatible systems. The canonical JevBench board is. JevBench by Benchmark Heaven is a benchmark for AI decision models, including Jev-compatible systems. It measures typed decision accuracy, probability calibration, latency and cost. Maintained by Benchmark Heaven, independently of TypeSafe AI. Current open-weights board · Hosted API board · Methodology · Data card · Benchmark comparison.

Quoted from the project's README. Read the full README.

Tagscommunity python benchmarks evaluation calibration

Works forJev for Evaluation Jev for Scoring

Used inJev for Software & DevTools

Highlights

Each complete entrant runs 534 frozen cases, including 220 hard decisions written by Claude Opus 5 and GPT-5.6 Sol and hashed before any system ran.
The JevBench Score is the geometric mean of chance-corrected Intelligence, Calibration, Speed and Cost, 25% each.
Jev 1.13.0 ranks second at 74.4 behind Hopper at 75.4 on the 66-row board, reading 950 input tokens per decision.
Cost is dollars per 1,000 decisions, not tokens: Jev's $0.0399 comes from 950 tokens at $0.042 per million.
Small models are option-order sensitive: reversing A and B order dropped open-alternative-jev from 72% to 21% on answer judging.

Watch out

MIT for the harness and 72 original public decisions; other artifacts keep their own licences. Not affiliated with TypeSafe AI, some self-hosted latency is adjusted by assumption, and held-out cases were exposed to the services measured.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

Hacker News: Show HN: JevBench, a reproducible benchmark for typed decision models

florianstandhar123 points · 34 commentsShow HN: JevBench, a reproducible benchmark for typed decision modelsRead the thread on Hacker News

Other listings of this project

Forks and copies of this project in the directory. This entry is the canonical one.

jevbench

Reproducible harness comparing Jev, Laya, two LLMs, and two BERT variants on SST-2, AG News, and Banking77 for accuracy, calibration, latency, throughput, and cost.

Jev Practices & Patterns#community#awesomejev#catalog-import

Community

dhruvmehra/jevbenchPython

From the Jev guides

Original write-ups on what this entry is for.

The trust ledger: measuring confidence your domain has earned

Oct 16, 2026

ASSAY-001 calibrated Jev on CLINC150 but found it overconfident on Banking77 — one audit, two verdicts. How calibration research measures the model-and-domain pair, discounts stated confidence, and signs the evidence.

Jev Benchmarks & Evaluations#calibration#evaluation#open-models

Read article

What the First Month of Jev Papers Found

Oct 14, 2026

More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.

Jev Practices & Patterns#research#evaluation#calibration

Read article

Measuring Jev: What Independent Benchmarks Actually Found

Sep 22, 2026

TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.

Jev Practices & Patterns#benchmarks#evaluation#calibration

Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai

JevBench — fstandhartinger

About this project

Highlights

Questions about this entry

Is this benchmark run by TypeSafe?

Are the latency and cost columns measured, or estimated?

Can a small model just game the option order?

Is the held-out half still held out?

What is the difference between the Capability and Composite scores?

How is this JevBench different from the one at jevbench.github.io?

What happens to a missing measurement on the JevBench board?

Reactions & coverage

Related terms

Other listings of this project

More Jev resources like this

From the Jev guides

From the Jev AI model community