User repository on GitHub
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Nudge the Judge: Uncovering Program Decision Hijacking in Real-World Jev Applications (arXiv 2610.18612)
An automated framework that traces adversarial context through Jev-backed program decisions to downstream effects. Across 228 real-world Jev applications and 2,273 test cases, 81.5% show decision flips, 98.6% with potential security impact, and Decision Flip Rate beats the strongest baseline by 50.4 points.
- Stars
- 2
- Language
- Python
- License
- MIT
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Defines Jev Program Decision Hijacking: attacker-controlled contexts redirect program decisions through schema-valid judgments.
- JevProbe traces adversarial influence from reachable context surfaces through program-consumed decisions to downstream consequences.
- 81.5% of 228 popular Jev applications showed decision flips across 2,273 test cases; 98.6% carried potential security impact.
- Improves macro-average Decision Flip Rate by 50.4 percentage points over the strongest adapted baseline; defenses helped little.
- Source code and the JevProbeBench dataset are public on GitHub, and findings were disclosed to maintainers.
Watch out
arXiv preprint posted October 13, 2026 (cs.CR, cs.AI). The 81.5% flip rate is self-reported via the authors' JevProbe framework on 228 selected popular applications, and the abstract does not describe how they were selected. Disclosure was pending, and potential downstream impact is not confirmed exploitation.
Questions about this entry
Short answers drawn from the linked project or page.
JPDH is the security risk this paper names: attacker-controlled contexts redirect the decisions of real-world Jev applications through judgments that remain schema-valid, so applications can act against their intent.
JevProbe evaluates 228 popular real-world Jev applications with 2,273 test cases, tracing adversarial influence from reachable context surfaces to program-consumed decisions and downstream consequences.
81.5% of the 228 applications exhibited decision flips, and 98.6% of those flips led to potential downstream security impacts, according to the paper.
It improves the macro-average Decision Flip Rate by 50.4 percentage points compared with the strongest adapted baseline, while the defenses the authors tested offered limited protection.
The authors argue existing evaluations focus on model-level attacks or predefined injection channels, leaving the attack reachability and downstream security consequences in real-world Jev applications underexplored.
The paper says the findings were responsibly disclosed to Jev application maintainers, and their responses were still pending when the preprint was posted.
Yes. The authors state that the source code of JevProbe and the JevProbeBench dataset are publicly available on GitHub at github.com/xiaoxiaohg8d/JevProbe-.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsHacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.
sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker NewsX: Secret detection with Jev, tested
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A paired, self-audited evaluation of an open-weight (Laya) and a hosted (Jev) System-1 decision model on 11 agent decision points: 7,283 base cases plus 6,640 robustness variants. Jev is significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
David-DL-Space/sys1-evalJavaScript
David-DL-Space/sys1-evalJavaScriptA controlled adversarial comparison of Jev against five GPT and GLM judges and two local classifiers under identical payloads and policies. Jev is the most robust on input moderation and the fastest overall, but detects only 62% of phishing emails and flags the most benign requests.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An empirical comparison of Jev as a typed judge for retrospective agent-trace security against four generative judges on 5,219 trajectories. Jev averages positive-class F1 77.8 versus 74.1 for GLM-5.2, leading on two of four benchmark collections.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
0GitHub stars
Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
JevAdvBench is a black-box adversarial benchmark for RLCD models such as Jev, with 812 typed questions over 66 scenarios and 9,744 single-edit attack variants. On jev-1.13.0 an appended unverified opinion flipped 12.1% of decisions and pushed 38% of confident answers below the review threshold.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
RLCDAlignBench measures Jev as a zero-shot detector of ten alignment failures across 44 benchmarks and five target models. A single generic question reaches median AUROC 0.886 and beats supervised baselines on most benchmarks at 63 times lower cost than LLM judges.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
sumleo/RLCDAlignBenchPython
sumleo/RLCDAlignBenchPythonFrom the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
A decision model cannot be talked into generating text, but the label is still a target. Why narrow, literal questions resist injection and why keeping the act in code is the real backstop.
Jev Practices & Patterns#guardrails#security#prompt-injection
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.