Article
CommunityJev Benchmarks & Evaluations
JevBreak: Your Model, Their Decision
arXiv paper presenting Slot Attack, a black-box attack that appends text to evidence to steer decision models toward an attacker-chosen wrong option. Evaluates seven decision models on 16 Decision Bench tasks across seven domains.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Slot Attack appends text assembled from named components to steer predictions toward an attacker-selected incorrect option.
- Reports 32-99% success on initially correct decisions across seven decision models.
- Uses only returned label probabilities, with a median of 17-72 queries per successful attack.
- Models with near-identical accuracy differ by about 20 percentage points in attack success.
- On about 20% of correct decisions for the JEV models, an attacker could force every available wrong answer.
Questions about this entry
Short answers drawn from the linked project or page.
Slot Attack is a black-box attack designed for decision models. It appends text assembled from named components to the evidence in order to steer predictions toward an attacker-selected incorrect option, relying solely on the label probabilities the model returns.
Seven decision models spanning hosted APIs, open-weight models and an encoder are evaluated on 16 tasks in seven domains from Decision Bench, targeting every wrong option across 240 text examples, or 990 targets per model.
The paper reports success rates of 32-99% on initially correct decisions. Accuracy and confidence do not reliably reveal the vulnerability, and models with near-identical accuracy on the sampled tasks differ by about 20 percentage points in attack success.
The method identifies effective component combinations with a median of 17-72 queries per successful attack, using only the label probabilities returned by the model.
No. The paper states that a confidence threshold does not reliably screen the attack out, and that accuracy and confidence do not reliably reveal the vulnerability.
Yes. The authors demonstrate forcing every available wrong answer on about 20% of correct decisions for the JEV models, 77% for gpt-6-luna and 98% for GLiNER2.5.
It concludes that a fixed answer set constrains a decision's form, not its source, and that robustness must be measured per model, per task and per version rather than assumed from output format.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting FlipJev, a fast black-box query-based red-teaming attack that finds short universal adversarial suffixes for Jev. It reports experiments on 12 datasets, 10,240 paired prompt-injection tests, and cross-task suffix transfer of 90-100%.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv paper presenting the first systematic bias and fairness audit of Jev, a typed-decision model for recruitment. It measures how demographic attributes and candidate cues shift screening scores, rankings, and shortlisting, against a noise floor, plus validity, latency, and cost.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#research
Community
arXiv paper evaluating 22 hosted decision models and two prompted general models as runtime classifiers for a multi-rule coding-agent policy engine. It reports recall, false-positive, latency, and cost results on a 2,643-call synthetic dataset with a sealed holdout.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Hugging Face dataset from TypeSafe AI pairing security incident telemetry with model decision outputs. Tabular and text parquet rows, English, 1K–10K records, loadable with Datasets, pandas, or Polars.
Jev Benchmarks & Evaluations#evaluation#benchmark#security
Community
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.