Article
CommunityJev Benchmarks & Evaluations
Escalate or Close: Fast-Tier Models on Held-Out Web Attack Families
arXiv study of two-tier web attack triage that holds seven attack families out of training, recalibration and thresholds, then compares zero-shot decision models including Jev with supervised detectors, a local language model, the OWASP Core Rule Set and an anomaly baseline.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Holds each of seven web attack families out of training, recalibration and thresholds, splicing payloads into parameters of real browsing traffic.
- Over 10 training runs, ModernBERT gives 37% to 60% of held-out path traversal a probability below 0.05 and escalates 50% to 81% within a 5% benign budget.
- Jev and Clef escalate at least 97.6% of every held-out family within the same budget, though their undisclosed training data may contain these public families.
- Jev escalates 37% of benign twins, exploit requests with the attack replaced by legitimate values.
- A search guided by Jev's probability gets 43% of held-out attacks past Jev within 300 queries; asked again, Jev escalates those requests in 60% of its answers.
Questions about this entry
Short answers drawn from the linked project or page.
In two-tier triage a fast model decides most web requests and escalates the rest to a slow tier, so any attack the fast tier closes is never examined by the slower, more capable tier.
Zero-shot decision models that return a probability for a typed question, including Jev and Clef, are compared with supervised detectors, a local language model, the OWASP Core Rule Set and an anomaly baseline.
Each of seven attack families is held out of training, recalibration and thresholds, and the payloads are spliced into parameters of real browsing traffic so the requests keep a realistic shape.
Over 10 training runs, ModernBERT gives 37% to 60% of held-out path traversal a probability below 0.05 and escalates only 50% to 81% of it within a 5% benign budget, against nearly all of the same requests when traversal is known.
Both escalate at least 97.6% of every held-out family within the 5% benign budget, and their budget decisions do not depend on the held-out family. The paper notes their undisclosed training data may contain these public families.
Request shape needs controlling: Jev escalates 37% of benign twins, which are exploit requests with the attack replaced by legitimate values, and detectors trained on older exploits learn what scanner requests look like.
A search guided by Jev's probability gets 43% of held-out attacks past Jev within 300 queries, and Jev's answers vary: asked again, it escalates those requests in 60% of its answers.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
YouTube: Kubernetes admission control with a decision model, fail-closed.
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
More Jev resources like this
arXiv paper evaluating Jev as a safety guardrail for a Qwen2.5-1.5B-based pipeline across 970 benchmark prompts, comparing an unfiltered LLM, Jev input filtering, and tiered input filtering with selective output review.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#guardrails
Community
arXiv paper presenting Slot Attack, a black-box attack that appends text to evidence to steer decision models toward an attacker-chosen wrong option. Evaluates seven decision models on 16 Decision Bench tasks across seven domains.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv paper presenting FlipJev, a fast black-box query-based red-teaming attack that finds short universal adversarial suffixes for Jev. It reports experiments on 12 datasets, 10,240 paired prompt-injection tests, and cross-task suffix transfer of 90-100%.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv paper testing whether Jev, a decision model that outputs judgment probabilities, works as a multilingual judge. It compares Jev with generative LLM judges across four annotated benchmarks, three evaluation formats, and up to 47 language varieties.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing hybrid decision models (HDM) with a self-routing block that estimates the value of thinking. Evaluates on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77, and MiniWoB; on JevBench HDM-4B scores 0.857 vs Jev's 0.851 at median 0.179s vs 0.652s.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting RouteAlign, a method that uses task requirements to structure model performance measurement and LLM service selection. Reports quality gains over open Jev-style decision backends at lower generation cost.
Jev Benchmarks & EvaluationsArticle#arxiv#routing#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.