CommunityJev Benchmarks & Evaluations
Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers (arXiv 2610.17820)
A controlled adversarial comparison of Jev against five GPT and GLM judges and two local classifiers under identical payloads and policies. Jev is the most robust on input moderation and the fastest overall, but detects only 62% of phishing emails and flags the most benign requests.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Sends every system the same payload under the same policy across four tasks: moderation, injection detection, routing, and phishing.
- Runs fifty attack variants from thirteen families against Jev, five GPT and GLM judges, and two local classifiers.
- Eight forgeries of Jev's structured input bypass none of its moderation decisions; a why-harm sentence bypasses 9%, against 56% for GPT-6 Astra.
- On phishing, forgeries bypass 19-29% of flagged emails, and unattacked Jev detects 62% against 77-81% for the GLM judges.
- Jev is the fastest judge but also flags the most benign requests, at 27%.
Watch out
arXiv preprint posted October 13, 2026 (cs.CR, cs.AI), 21 pages and 10 figures. The comparison is the authors' own harness on four public-data tasks and thirteen specific attack families, so results may not generalize to other policies or payloads; the vendor supplied no adversarial numbers to check against.
Questions about this entry
Short answers drawn from the linked project or page.
Jev generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer, while the GPT and GLM judges in the comparison generate outputs.
The harness runs four public-data tasks covering input moderation, injection detection, routing, and phishing detection, attacked with fifty variants from thirteen families under one shared policy.
On input moderation, eight forgeries of Jev's structured input bypassed none of its decisions, while a sentence asking only why someone might make a harmful request bypassed 9% of its blocks.
The same forgeries bypassed 19-29% of the phishing emails it flags, and even without attack Jev detected only 62% of phishing emails, against 77-81% for the GLM judges.
Jev was the fastest judge tested, but it also flagged the most benign requests, at 27%, a false-positive trade-off the authors report alongside its robustness results.
They failed differently from the others: a statement drawn from training examples made Llama Guard 3 miss 62% of the phishing emails it had previously flagged.
The authors conclude Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment, such as phishing detection.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsHacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.
sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker NewsX: Secret detection with Jev, tested
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
David-DL-Space/sys1-evalJavaScript
David-DL-Space/sys1-evalJavaScript2GitHub stars
An automated framework that traces adversarial context through Jev-backed program decisions to downstream effects. Across 228 real-world Jev applications and 2,273 test cases, 81.5% show decision flips, 98.6% with potential security impact, and Decision Flip Rate beats the strongest baseline by 50.4 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
xiaoxiaohg8d/JevProbe-Python
xiaoxiaohg8d/JevProbe-PythonAn empirical comparison of Jev as a typed judge for retrospective agent-trace security against four generative judges on 5,219 trajectories. Jev averages positive-class F1 77.8 versus 74.1 for GLM-5.2, leading on two of four benchmark collections.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
JevAdvBench is a black-box adversarial benchmark for RLCD models such as Jev, with 812 typed questions over 66 scenarios and 9,744 single-edit attack variants. On jev-1.13.0 an appended unverified opinion flipped 12.1% of decisions and pushed 38% of confident answers below the review threshold.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
RLCDAlignBench measures Jev as a zero-shot detector of ten alignment failures across 44 benchmarks and five target models. A single generic question reaches median AUROC 0.886 and beats supervised baselines on most benchmarks at 63 times lower cost than LLM judges.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
sumleo/RLCDAlignBenchPython
sumleo/RLCDAlignBenchPythonJevOut shows short background context additions can flip correct decisions in four systems including Jev across seven datasets. Within 64 accepted evaluations per decision, additions redirect 61.4%-73.2% of initially correct decisions toward a wrong option fixed in advance.
Jev Practices & PatternsArticle#community#research#arxiv
Community
xzx34/JevOutPython
xzx34/JevOutPythonFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.