Article
CommunityJev Benchmarks & Evaluations
Jev as an LLM Safety Guardrail: An Empirical Study of Input Filtering, Selective Output Review, and Over-Refusal
arXiv paper evaluating Jev as a safety guardrail for a Qwen2.5-1.5B-based pipeline across 970 benchmark prompts, comparing an unfiltered LLM, Jev input filtering, and tiered input filtering with selective output review.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Compares three configurations on 970 prompts: unfiltered LLM, Jev input filtering, and tiered filtering with selective output review.
- Harmful-prompt answers drop from 527/720 (73.2%) unfiltered to 17/720 (2.4%) with Jev input filtering.
- Benign-prompt refusals rise from 3/250 (1.2%) to 27/250 (10.8%) under input filtering.
- Tiered filtering yields 40/720 (5.6%) harmful answers and 10/250 (4.0%) benign refusals, with 79/970 (8.1%) prompts needing output review.
- Joint threshold optimization lifts macro-F1 from 0.9351 to 0.9512 while cutting harmful answers and review frequency at the same benign-refusal rate.
Questions about this entry
Short answers drawn from the linked project or page.
The evaluation places Jev as a safety guardrail in front of a Qwen2.5-1.5B-based pipeline, comparing an unfiltered LLM, Jev input filtering, and tiered Jev input filtering with selective output review.
970 benchmark prompts in total: 720 labeled harmful (520 AdvBench and 200 XSTest-unsafe) and 250 XSTest-safe prompts used to measure over-refusal.
Across the 720 harmful prompts, the fraction recorded as answered falls from 527/720 (73.2%) without filtering to 17/720 (2.4%) with Jev input filtering.
On the 250 XSTest-safe prompts, non-answer/refusal increases from 3/250 (1.2%) without filtering to 27/250 (10.8%) with Jev input filtering.
It yields 40/720 (5.6%) harmful-prompt answers and 10/250 (4.0%) benign-prompt refusals, with 79/970 (8.1%) of prompts requiring output review.
Joint optimization improves macro-F1 from 0.9351 to 0.9512 while reducing harmful-prompt answers and output-review frequency at the same benign-refusal rate.
Matched-budget comparisons show modest additional gains over a retuned single gate on the evaluation data, but those gains are not consistently observed on a held-out split.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
arXiv paper presenting Slot Attack, a black-box attack that appends text to evidence to steer decision models toward an attacker-chosen wrong option. Evaluates seven decision models on 16 Decision Bench tasks across seven domains.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv study of two-tier web attack triage that holds seven attack families out of training, recalibration and thresholds, then compares zero-shot decision models including Jev with supervised detectors, a local language model, the OWASP Core Rule Set and an anomaly baseline.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting FlipJev, a fast black-box query-based red-teaming attack that finds short universal adversarial suffixes for Jev. It reports experiments on 12 datasets, 10,240 paired prompt-injection tests, and cross-task suffix transfer of 90-100%.
Jev Benchmarks & EvaluationsArticle#arxiv#security#evaluation
Community
arXiv paper examining numerical nondeterminism in vLLM Split-K LoRA kernels and introducing a deterministic Triton Split-K implementation, measured on GSM8K, MBPP, and Jev-style structured toxicity decisions over Civil Comments.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.