CommunityJev Benchmarks & Evaluations
JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges (arXiv 2609.34862)
An empirical comparison of Jev as a typed judge for retrospective agent-trace security against four generative judges on 5,219 trajectories. Jev averages positive-class F1 77.8 versus 74.1 for GLM-5.2, leading on two of four benchmark collections.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Compares Jev with four generative judges on four benchmark collections totaling 5,219 agent trajectories.
- Jev reaches benchmark-averaged positive-class F1 77.8, against 74.1 for the strongest generative configuration, GLM-5.2.
- Valid-result coverage is 95.5% for Jev and 94.4% for the generative judges under a common risk rubric.
- Jev leads on ATBench500 and MCPHunt, while GLM leads on R-Judge and TraceSafe.
- Median successful-call latency is 0.99 seconds, with estimated token cost of $0.000195 per valid judgment.
Watch out
arXiv preprint posted September 28, 2026 (cs.CR) by Zhiqiang Wang and Yichao Gao. The comparison is self-run on four public benchmark collections under one rubric, so results may not transfer to other trace formats or policies; the authors position Jev as an economical screening signal, with trade-offs in precision and recall.
Questions about this entry
Short answers drawn from the linked project or page.
The authors test JEV, a typed decision model, as an alternative to generative judges for retrospective security classification of tool-using agent traces.
It covers four benchmark collections totaling 5,219 trajectories, all judged under a common risk rubric with behavior-level labels.
JEV attains a benchmark-averaged positive-class F1 of 77.8, compared with 74.1 for the strongest generative configuration, GLM-5.2.
Performance varied by dataset: JEV led on ATBench500 and MCPHunt, while the generative GLM judges led on R-Judge and TraceSafe.
Across the four benchmarks, JEV's median successful-call latency is 0.99 seconds, and estimated token cost averages $0.000195 per valid judgment.
Valid-result coverage is 95.5% for JEV and 94.4% for the generative judges, a small gap the authors note alongside the latency and cost differences.
They support JEV as an economical screening signal for agent-trace security, acknowledging trade-offs in precision and recall rather than proposing it to replace generative judges.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsHacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.
sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker NewsX: Secret detection with Jev, tested
More Jev resources like this
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
David-DL-Space/sys1-evalJavaScript
David-DL-Space/sys1-evalJavaScriptA benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPython2GitHub stars
An automated framework that traces adversarial context through Jev-backed program decisions to downstream effects. Across 228 real-world Jev applications and 2,273 test cases, 81.5% show decision flips, 98.6% with potential security impact, and Decision Flip Rate beats the strongest baseline by 50.4 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
xiaoxiaohg8d/JevProbe-Python
xiaoxiaohg8d/JevProbe-PythonA controlled adversarial comparison of Jev against five GPT and GLM judges and two local classifiers under identical payloads and policies. Jev is the most robust on input moderation and the fastest overall, but detects only 62% of phishing emails and flags the most benign requests.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
JevAdvBench is a black-box adversarial benchmark for RLCD models such as Jev, with 812 typed questions over 66 scenarios and 9,744 single-edit attack variants. On jev-1.13.0 an appended unverified opinion flipped 12.1% of decisions and pushed 38% of confident answers below the review threshold.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
RLCDAlignBench measures Jev as a zero-shot detector of ten alignment failures across 44 benchmarks and five target models. A single generic question reaches median AUROC 0.886 and beats supervised baselines on most benchmarks at 63 times lower cost than LLM judges.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
sumleo/RLCDAlignBenchPython
sumleo/RLCDAlignBenchPythonFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.