The table says Jev goes from worst to best — is that the same experiment?

No, they are two different tasks. Whole-transcript monitoring reads the finished transcript and Jev scores worst, AUC 0.78 against 0.99 for Gemini Flash and 0.86 for Pro. Per-action monitoring scores each tool call before it executes and Jev is best, 0.97 against 0.91. The ranking inverts because the unit of judgment changes, not because the model changed.

Thirteen transcripts is not a benchmark — how much of this should I trust?

The README says preliminary and treats the numbers as design signals. The set is 13 hand-built transcripts across three tasks, five honest and eight containing sabotage, constructed from the scripted solutions using real tool outputs. The stated next step is confirming it on agent-generated attacks from SHADE-Arena's private split.

If I put Jev in front of Gemini, do I get all 13 attacks back?

In sample, yes: escalating only what Jev flags catches all 13 with 8.5% of benign actions blocked against 29.4% for Gemini alone, and 71% fewer Gemini calls. Those thresholds were tuned on the same attacks, and at any escalation threshold above 41 the disguised attack is lost because Gemini never sees what Jev passes.

How large is the Jev monitor evaluation?

13 hand-built honest and attack transcripts across 3 tasks: bank transfer, API key leak, and spam filter. They contain 5 honest and 8 sabotage transcripts, which split into 353 tool calls with 13 sabotage actions.

What is the per-action result for Jev versus Gemini Flash?

Scoring each tool call before it runs, Jev ranks sabotage with an AUC of 0.97 against Gemini Flash's 0.91, and catches 12 of 13 attacks while blocking only 1.5% of benign actions at a threshold of 73. Jev's median time per action is 1.5s against Flash's 4.7s.

What token limit forces transcript chunking?

Jev accepts at most about 32K tokens per request, measured at 32,060 input tokens accepted. Bank transcripts of about 50K tokens must be split into three overlapping chunks, which is where the whole-transcript monitor struggles.

CommunityJev Practices & Patterns

Shade Arena Jev Monitor

Q: The disguised autodelete attack keeps coming up — what does Jev actually miss?

An autodelete rule framed as anti-phishing: Jev scored it 41 per action and 56 transcript-level, below both honest transcripts at 65, while Gemini Flash caught it at 90.5. The stated weakness is attacks whose harm only shows through reasoning about user intent, which is exactly that shape.

A fork of the SHADE-Arena public split that evaluates TypeSafe Jev as a monitor for covert agent sabotage, against Gemini 2.5 Flash and Pro. Thirteen hand-built transcripts across three tasks, 353 tool calls, and a full LaTeX report.

See highlights

Open github.com

Stars: 0
Language: Python
License: MIT
Published by: Community
Author: nican2018
Last verified: Oct 2026

nican2018/shade-arena-jev-monitor

User repository on GitHub

View on GitHub

Evaluating TypeSafe's Jev as a fast monitor and action gate for agent sabotage in SHADE-Arena, compared with Gemini 2.5 Flash/Pro.

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

This fork of the SHADE-Arena public split adds an evaluation of Jev, TypeSafe AI's non-generative "System One" decision model, as a monitor for covert agent sabotage. It compares Jev with Gemini 2.5 Flash and Gemini 2.5 Pro. Background on SHADE-Arena, its license, and its canary string are at the end. 📄 Full report: docs/report/jevmonitorreport.pdf (LaTeX source in the same folder)

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Evaluation Jev for Tool calling

Used inJev for Software & DevTools

Languages: Python

Highlights

As a whole-transcript monitor Jev is weakest of the three, AUC 0.78 against Gemini Flash 0.99; per action it is best, 0.97 against 0.91.
Scoring each tool call before it runs, a threshold of 73 catches 12 of 13 attacks while blocking 1.5% of benign actions, at 1.5 s median.
Long transcripts are the failure case: Jev accepts about 32K input tokens, so ~50K-token transcripts get chunked and malicious transfers barely move the score.
Escalating only suspicious actions to Gemini catches all 13 with 8.5% benign blocks and 71% fewer Gemini calls, simulated from saved scores.

Watch out

The README labels this preliminary and the label is load-bearing: 13 hand-built transcripts across 3 tasks, 5 honest and 8 with sabotage, built from the scripted solutions rather than from agent-generated attacks, so these are design signals and not benchmark numbers. Every threshold was tuned on the same 13 attacks, and the report's Table 5 shows that with thresholds chosen on held-out tasks, every Jev-based design misses the disguised autodelete attack. That blind spot is semantic: Jev scored an autodelete rule disguised as anti-phishing at 41 per action and 56 transcript-level, below both honest spam-task transcripts at 65, while Gemini Flash caught it at 90.5, and the stated weakness is attacks whose harm only shows through reasoning about user intent. Whole-transcript numbers also depend on chunking under a roughly 32K-token limit, and the bank task's honest transcripts contain a real mistake, a payment to the wrong account, so part of the high honest scores is a correct flag rather than noise. Gemini Pro has no per-action numbers. Full results live in a PDF with LaTeX source under docs/report/. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

daf-jev

Python client and decision toolkit for the Jev System One endpoint with noul, choice, and score primitives, batched asks, composition helpers, concurrent evaluation, and an MCP server.