Skip to content

Jev directory

Evaluating Jev as a Prompt-Injection Guardrail

An independent evaluation of TypeSafe's Jev as a prompt-injection detector: 6,991 Jev calls across six tests with three comparison detectors, publishing all code, inputs, per-prompt results and figures, plus a written analysis.
Stars
1
Language
Python
Reading time
2 min
Published by
Community
Author
Ruchith1018
Last verified
Oct 2026

Independent evaluation of TypeSafe AI's Jev decision model as a prompt-injection guardrail: 6,991 calls, six tests, code, results and figures.

Forks
0
Last pushed
Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Highlights

  • 6,991 Jev calls across six tests, with three comparison detectors run on the same inputs; all code and results are public.
  • Jev ranks attacks above safe input with AUC 0.986 on deepset, vs 0.882 (ProtectAI) and 0.910 (DeepSeek Flash).
  • At the default 0.5 cut-off: 58.6% of deepset attacks caught, and 0 of 399 safe prompts blocked.
  • A tuned 0.12 cut-off caught 81.7% on held-out deepset data but blocked 74.7% of clean documents — no transfer across input kinds.
  • Strongest result is over-blocking: 0.6% of 339 trigger-word-heavy safe prompts flagged vs 43.4% for ProtectAI.

Watch out

README and GitHub metadata reviewed 2026-10-17; the digest truncates the findings table inside its last row, so the closing conclusion is only partial. The repository declares no license, and the author states the work is not affiliated with or endorsed by TypeSafe AI.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

YouTube: Kubernetes admission control with a decision model, fail-closed.

Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.

2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker News

X: Jev and Laya on security queries

X: Testing Jev for prompt injection

From the Jev guides

Original write-ups on what this entry is for.

From the Jev AI model community

Posts from builders shipping with Jev right now.

Follow @typesafeai