Why is robustness hard to measure for typed decision models?

Identical requests can return different answers, most available labels come from the model itself, and the API preprocesses each request out of view, so generative adversarial benchmarks do not apply directly.

How does JevAdvBench score an attacked decision?

It scores each attacked decision against the model's own clean decision instead of labels, and reads the result against the change caused by an identical re-run.

What does the benchmark contain?

JevAdvBench includes 812 typed questions across 66 scenarios and a black-box attack suite of 9,744 single-edit variants, each editing one part of a request.

How do the authors verify that an edit reached the model?

They use billed input tokens to confirm that each single-edit variant actually reached the model through the API's out-of-view preprocessing.

How much did rewording move jev-1.13.0's decisions?

Rewording stayed within 1.2 percentage points of the re-run baseline, and fields outside the schema never reached the model, according to the paper.

How effective was the appended-opinion attack?

One unverified opinion appended to the state flipped 12.1% of decisions, statistically tied with the strongest injected command at 10.1%.

What does the paper recommend for applications built on Jev?

Applications built on RLCD models should treat the state as untrusted, argued input, since a single edit pushed 38% of confident answers below the human-review threshold.

CommunityJev Benchmarks & Evaluations

JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models (arXiv 2609.31142)

JevAdvBench is a black-box adversarial benchmark for RLCD models such as Jev, with 812 typed questions over 66 scenarios and 9,744 single-edit attack variants. On jev-1.13.0 an appended unverified opinion flipped 12.1% of decisions and pushed 38% of confident answers below the review threshold.

See highlights

Open arxiv.org

Format: Article
Published by: Community
Last verified: Oct 2026

Tagscommunity research arxiv security adversarial calibration

Works forJev for Evaluation Jev for Decision gating

Used inJev for Security & Privacy Jev for Software & DevTools

Highlights

First adversarial benchmark for RLCD models, per the authors: 812 typed questions over 66 scenarios.
Black-box suite of 9,744 single-edit variants, each editing one part of a request, verified via billed input tokens.
Scores attacked decisions against the model's own clean answer and an identical re-run rather than against labels.
An appended unverified opinion flipped 12.1% of decisions, statistically tied with the strongest injected command at 10.1%.
That edit pushed 38% of confident answers below the 0.8 confidence threshold that routes them to human review.

Watch out

arXiv preprint posted September 25, 2026 (cs.CR, cs.AI, cs.CL), 33 pages and 19 tables. Findings are the authors' black-box measurements on jev-1.13.0 using the label-free methodology they designed; the abstract does not report other model versions, and confidence thresholds are specific to the configurations tested.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

YouTube: Kubernetes admission control with a decision model, fail-closed.

Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.

2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker News

Hacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.

sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker News

X: Secret detection with Jev, tested

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers (arXiv 2610.17820)

A controlled adversarial comparison of Jev against five GPT and GLM judges and two local classifiers under identical payloads and policies. Jev is the most robust on input moderation and the fastest overall, but detects only 62% of phishing emails and flags the most benign requests.