Article
CommunityJev Benchmarks & Evaluations
FlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision Models
arXiv paper presenting FlipJev, a fast black-box query-based red-teaming attack that finds short universal adversarial suffixes for Jev. It reports experiments on 12 datasets, 10,240 paired prompt-injection tests, and cross-task suffix transfer of 90-100%.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Black-box, query-based attack that calls no attacker language model.
- Experiments span 12 datasets with different data types and application scenarios.
- No prompt-injection strategy changed a correct answer across 10,240 paired tests.
- One suffix per dataset flips 43% of unseen documents.
- A suffix generated from one task flips 90-100% of samples in other tasks.
Questions about this entry
Short answers drawn from the linked project or page.
It is a quick black-box, query-based attack that searches for short adversarial suffixes. It exploits the speed of typed decision models, which answer in a single fast and cheap query, rather than calling an attacker language model.
No. The paper states the attack calls no attacker language model, so it operates purely through black-box queries against the target typed decision model.
According to the paper, Jev resists common prompt injections: no strategy changed a correct answer across 10,240 paired tests.
One suffix per dataset flips 43% of unseen documents, and the authors report that a suffix generated from one task can flip 90-100% of the samples in other tasks.
The experiments cover 12 datasets with different data types and application scenarios, and also test whether suffixes transfer across tasks.
The paper reports that the attack finds short suffixes which flip confident answers within minutes, in contrast to prompt-injection strategies that did not change answers.
They argue that evaluation of typed decision models must include strict adaptive attacks to fully reveal their robustness, because typed models can serve many tasks and suffixes transfer between them.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing SatOps-Decision, a locally adapted Laya encoder with its native decision head that selects among supplied satellite-operations answers in one forward pass, plus a benchmark of 1,024 held-out questions with latency and accuracy comparisons.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
arXiv study evaluating Jev as a decision reader of truncated LLM reasoning across seven reasoners and five multiple-choice benchmarks, comparing accuracy and estimated cost against forced answering at budgets from 500 to 16,000 tokens.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.