User repository on GitHub
Blind security benchmarks for Jev, TypeSafe's System One model: prompt injection and vulnerable code detection, built on jev-go
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Practices & Patterns
Jev Sec Bench
Blind security benchmarks for Jev on public corpora: 662 prompt-injection messages and 200 matched vulnerable-code pairs, with raw per-sample results and a TUI dashboard.
- Stars
- 3
- Language
- Go
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- Gaurav-Gosain
- Last verified
- Sep 2026
About this project
Blind security benchmarks for Jev, TypeSafe's System One model, built on jev-go. Jev does not generate text. It reads a state and returns typed judgments with calibrated probabilities, which is the shape a guardrail actually needs: a number your code can threshold, not a paragraph you have to parse. Two benchmarks, both blind, both on public corpora: prompt injection, on all 662 labelled messages in deepset/prompt-injections. vulnerable code, on 200 matched pairs from CyberNative/CodeVulnerabilitySecurityDPO.
Quoted from the project's README. Read the full README.
Highlights
- Prompt-injection run at a plain 0.50 cut: 96.5% accuracy, 95.1% recall, ROC-AUC 0.9927, ECE 0.0588, 22.7s for 662 messages at p50 325 ms.
- Adding deployment context to the state raised injection recall from 74.9% to 95.1% without any threshold tuning.
- In 200 matched code pairs the vulnerable half scored above its secure twin 178 times (89.0%), including 43/46 SQL injection pairs.
- The audit found at least 38% of apparent false positives are corpus label errors, counting only what a regex can prove.
Quickstart
bash
export TYPESAFE_API_KEY=YOUR_API_KEY
go run ./cmd/jev-sec-bench -bench all
go run ./cmd/jev-tuiWatch out
MIT-licensed. Needs Go and a TYPESAFE_API_KEY; both corpora are public and may be in training data, results are a single run, and thresholds should be validated on your own labelled data.
Questions about this entry
Short answers drawn from the linked project or page.
Plain 0.50 cut, no threshold tuning: 662 messages with 263 injections, 10 false positives and 13 false negatives. But the headline depends on passing deployment context in the state. Without it accuracy is 89.7% and recall falls to 74.9%, because that corpus assumes a reader assistant for a German news publisher, where a topic hijack counts as subverting it.
Treat it as a floor. In 200 matched pairs the vulnerable half scored above its secure twin 178 times (89.0%), and both halves share topic, language and style so surface features cannot carry that. An audit found at least 38% of apparent false positives are corpus label errors, counting only what a regex can prove.
Reading is offline. go run ./cmd/jev-tui draws the committed per-sample JSON from results/, and -shot with -svg writes one frame as a picture, so the screenshots are generated by the same code that produced the numbers and cannot drift from them. Only re-running inference needs TYPESAFE_API_KEY exported.
Probably not as-is. The injection corpus is specific to one German news assistant and skews German, the code corpus is synthetic, and both are public so they may have leaked into training data, Jev included. It was a single run with no repeats, so small per-class cells carry real variance. Validate thresholds on your own labelled data.
With Go installed, `go run ./cmd/jev-sec-bench -bench all` runs the benchmarks and `go run ./cmd/jev-tui` reads the committed JSON back as a dashboard with overview, injection, code and live tabs. A TypeSafe key is only needed for new runs and the live feed.
Adding one line telling Jev what the assistant is for raised injection accuracy from 89.7% to 96.5% and recall from 74.9% to 95.1%, while ROC-AUC barely moved from 0.9846 to 0.9927. The lesson is that a guardrail which only sees the message is guessing at the policy.
The 200 matched pairs share topic, language and style, with one half carrying a vulnerability; the halves are shuffled apart and scored independently, and Jev is never told the vulnerability class or that pairs exist. The vulnerable half ranked above its secure twin in 178 of 200 pairs.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
A public benchmark comparing Jev with Claude Haiku 4.5 on whether an email agent should click a link, over 2,000 PhishNChips emails, with calibration, latency, cost, and five signal questions.
Jev Practices & Patterns#community#python#security
Community
anisselbd/jev-phishing-benchPython
anisselbd/jev-phishing-benchPythonA measurement harness that sends each file snippet to Jev as one Noul question asking whether the content contains a usable secret credential, then scores the returned probability against the expected label.
Jev Practices & Patterns#community#python#security
Community
teyhouse/jev-secret-detectionPython
teyhouse/jev-secret-detectionPythonSixteen thousand calls against gpt-5.4-mini and gpt-5.6-luna on four public datasets plus production pipeline decisions, with a threshold procedure and an explicit filter-not-replacement verdict.
Jev Practices & PatternsArticle#community#benchmarks#evaluation
Community
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonUnofficial companion to dspy-typesafeify that records and replays TypeSafe calls to measure calibration, selective risk, confidence-gated abstention, latency, tokens and modeled cost for Jev decisions.
Jev Practices & Patterns#community#python#evaluation
Community
jmanhype/jev-dspy-labPython
jmanhype/jev-dspy-labPythonAn independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.
Jev Practices & Patterns#community#python#routing
Community
FirasSX914/JanusPython
FirasSX914/JanusPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.