Is beating GPT-5.4 on agent failures a like-for-like comparison?

Only partly. Who and When are constrained-choice for Jev, which picks an agent or step from the options listed in the trace, while the LLM baselines free-generate. So the like-for-like axis is What: error-type F1 over the same 17-code taxonomy, where Jev scores 23.7 against the next best 22.2.

Can the model just read the answer key off the input?

Ground-truth labels never enter Jev's input, and tests/test_leakage.py enforces that. Jev is scored with the official whowhen_eval scorer pinned at commit 14369dcb, so its rows are directly comparable with the paper's baselines rather than scored by a harness that could flatter it.

Are these failures real incidents or synthetic ones?

Injected. Who&When Pro failures come from a controlled pipeline that replays a successful run and inserts one error, so they are not natural production incidents. The README is explicit that this is not a leaderboard submission, and it also carries a separate zero-shot-router pilot on RouterArena that came out negative.

What does a full run cost, and what do I need before I start?

About $1.28 for all 6,257 traces, because Jev bills input only and output tokens are free. Setup is uv plus a TYPESAFE_API_KEY, and the dataset is not redistributed: you curl a pinned text.jsonl and taxonomy.yaml of about 72 MB from Hugging Face at a fixed revision. jevbench estimate prices a sample offline before you spend.

What are the three questions Jev answers per trace?

For each failed multi-agent run it makes three typed choice decisions: the responsible agent, the decisive step, and the error mode, each returning a calibrated probability distribution over the allowed set.

How do I run it, and how do I avoid overspending?

Set up with uv and a TYPESAFE_API_KEY, fetch the pinned dataset, then run jevbench sample --n 6257, jevbench estimate for an offline cost estimate, jevbench run and jevbench report. Runs are resumable and record failures rather than dropping them.

What is the code license, and is the dataset included?

The code is Apache-2.0. The Who&When Pro dataset keeps its own CC-BY-4.0 license and is not redistributed, so you curl a pinned text.jsonl and taxonomy.yaml of about 72 MB from Hugging Face at a fixed revision.

Jev directory

CommunityJev Practices & Patterns

Jev Agent Failure Benchmark

A benchmark that runs Jev on all 6,257 text traces of Who&When Pro to attribute agent failures, scoring it with the official pinned scorer against the paper's GPT-5.4, Claude Sonnet 4.6, GLM-5, and Qwen3.5-122B baselines.

See highlights Open github.com

Stars: 4
Language: Python
License: APACHE-2.0
Reading time: 3 min
Published by: Community
Author: TokenTrim
Last verified: Sep 2026

TokenTrim/jev-agent-failure-benchmark

Organization repository on GitHub

View on GitHub

Benchmarking Jev (Typesafe.ai) against a strong LLM on the Who&When Pro agent-failure-attribution benchmark (text subset).

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Can a fast, cheap decision model find what broke an AI agent as well as a frontier LLM? This benchmarks Jev (Typesafe.ai) on the text subset of Who&When Pro, an agent-failure-attribution benchmark: given a failed multi-agent run, predict the responsible agent, the decisive step, and the error type. On all 6,257 text traces, Jev outperforms GPT-5.4 on every axis — for $1.28 total (Jev bills input only; output tokens are free).

Quoted from the project's README. Read the full README.

Tagscommunity python benchmarks evaluation agents

Works forJev for Evaluation Jev for Scoring

Used inJev for Software & DevTools

Highlights

Jev outperformed GPT-5.4 on every axis at about $1.28 total: Who 73.4, When 76.4, and error-type F1 23.7, the best in the field.
Jev answers three typed Choice questions per trace (responsible agent, decisive step, error mode), each with a calibrated probability distribution.
The official whowhen_eval scorer is pinned at commit 14369dcb so rows are directly comparable with the paper's numbers.
Who and When are constrained-choice for Jev while the LLM baselines free-generate, so the like-for-like axis is What, the error type over a shared taxonomy.
Ground-truth labels never enter Jev's input, and a leakage test enforces it; runs are resumable and record failures rather than dropping them.

Quickstart

bash

uv venv && uv pip install -e ".[dev]"
mkdir -p data
REV=0bd196c8a040841c4ae167ab33cc8151de246f1f
curl -L "https://huggingface.co/datasets/Leoxx/whowhen_pro/resolve/$REV/data/text.jsonl" -o data/text.jsonl
curl -L "https://huggingface.co/datasets/Leoxx/whowhen_pro/resolve/$REV/taxonomy.yaml" -o data/taxonomy.yaml
jevbench sample --n 300 --out results/run/sample.json
jevbench run --sample results/run/sample.json
jevbench report

Watch out

Apache-2.0 code; the CC-BY-4.0 Who&When Pro dataset is not included and must be fetched, failures are injected by a controlled pipeline rather than being natural incidents, and running needs Python with uv plus a TYPESAFE_API_KEY.

Questions about this entry

Short answers drawn from the linked project or page.

From the Jev guides

Original write-ups on what this entry is for.

What the First Month of Jev Papers Found

Oct 14, 2026

More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.

Jev Practices & Patterns#research#evaluation#calibration

Read article

Choosing an Open Decision Model

Oct 10, 2026

Interface, size, license, claims — in that order. A selection guide across the system-one catalog, from a 706K form scorer to 27B Jev students.

Jev Practices & Patterns#open-models#evaluation#benchmarks

Read article

Measuring Jev: What Independent Benchmarks Actually Found

Sep 22, 2026

TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.

Jev Practices & Patterns#benchmarks#evaluation#calibration

Read article

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Jev Agent Failure Benchmark

About this project

Highlights

Quickstart

Questions about this entry

Is beating GPT-5.4 on agent failures a like-for-like comparison?

Can the model just read the answer key off the input?

Are these failures real incidents or synthetic ones?

What does a full run cost, and what do I need before I start?

What are the three questions Jev answers per trace?

How do I run it, and how do I avoid overspending?

What is the code license, and is the dataset included?

Related terms

More Jev resources like this

From the Jev guides

From the Jev AI model community