What is in this Space's static snapshot?

A build-time rendering of the JevBench v1.6.1 table fetched on 2026-10-05, stored in index.html so it stays readable without JavaScript, with snapshot.json holding the feed exactly as fetched.

Who maintains JevBench?

Benchmark Heaven maintains it independently of TypeSafe AI; Jev is one of the systems evaluated, not the benchmark owner, and the site disambiguates itself from a separate metamorphic coherence benchmark.

What does the Capability Score measure?

It is the arithmetic mean of Intelligence and Calibration, with ranks limited to systems inside the official cost cap of $0.0646 per 1,000 decisions and the 1.23-second adjusted median-latency cap.

How is the Composite Score different?

It is a weighted harmonic mean of Intelligence, Calibration, Speed and Cost multiplied by squared penalties when Intelligence is below its floor or Speed or Cost is below 50; equal 25% weights do not make it an arithmetic average.

Which axes does JevBench measure?

Typed decision accuracy, probability calibration, latency and cost, evaluated for AI decision models including Jev-compatible systems; only system-level aggregates are published, with no item-level data.

What are the documented limitations?

Finite task coverage, sealed inputs that prevent complete independent reruns, potentially opaque hosted endpoint versions, modeled cost, runtime-dependent latency, mandatory disclosure for training on the public split, and no reliability guarantee on your workload.

Where are the method and the raw data?

The README links the fstandhartinger/jevbench GitHub repo for method and code, a results API and manifest, and the benchmarkheaven/jevbench dataset holding one row per system in data/leaderboard.csv.

CommunityJev Demos & Experiments

JevBench Leaderboard

Benchmark Heaven's static snapshot of the JevBench v1.6.1 decision-model board: capability and composite scoring over typed-decision accuracy, calibration, latency, and cost, readable without JavaScript.

See highlights

Open huggingface.co

Format: Playground
Published by: Community
Author: Benchmark Heaven
Last verified: Oct 2026

Tagscommunity playground demo benchmark leaderboard

Works forJev for Evaluation Jev for Ranking

Used inJev for Software & DevTools Jev for Data & Analytics

Platforms: Browser
Capabilities: Open sourceRuns locally

Highlights

Preserves JevBench v1.6.1 as fetched on 2026-10-05, before the 2026-10-06 publication and its addenda; the live board carries later additions.
Capability Score averages Intelligence and Calibration, ranked inside a $0.0646 per 1,000 decisions cost cap and a 1.23-second median-latency cap.
Composite Score is a weighted harmonic mean of all four axes with squared penalties below configured floors; equal 25% weights are not an arithmetic average.
Missing measurements are null, never zero, and no third-party benchmark numbers are included; only system-level aggregates are published.
The README positions Jev as one evaluated system, states Benchmark Heaven is independent of TypeSafe AI, and links method and code on GitHub.

Watch out

A preserved snapshot rather than the live board: the README directs you to benchmarkheaven.com for later dated reruns and presentation revisions, warns against comparing scores across method versions, and notes limitations — sealed inputs prevent complete independent reruns, cost is modeled from the published basis, and latency depends on runtime and measurement location.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

Playground

More Jev resources like this

Jev-Mem

The project page for Jev-Mem, a system-one-controlled agentic memory design from UT Dallas researchers: it reports 0.777 on LoCoMo, 6.6x faster memory construction, and 36.7% lower query latency than the fastest baseline.