User repository on GitHub
Measures how well TypeSafe's RLCD-Jev model spots real secret credentials in file snippets
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Practices & Patterns
Jev Secret Detection
A measurement harness that sends each file snippet to Jev as one Noul question asking whether the content contains a usable secret credential, then scores the returned probability against the expected label.
- Stars
- 2
- Language
- Python
- Reading time
- 3 min
- Published by
- Community
- Author
- teyhouse
- Last verified
- Sep 2026
About this project
Measures how well TypeSafe's Jev model spots real secret credentials in file snippets. Each test case is sent to Jev as a single Noul question ("does content contain a real secret credential that someone reading it could use?"), and the returned probability is compared with the expected label. There is deliberately no regex matching or provider verification. The goal is to score Jev itself.
Quoted from the project's README. Read the full README.
Highlights
- The default 100-case set splits 50/50 across vendor tokens, embedded credentials, generic secrets, passwords, connection strings, and private keys.
- The report lists accuracy at a 0.5 threshold, AUC, recall, precision, mean noul per class, Brier score, and how many cases land in the 0.3-0.7 review band.
- Latency output includes round trip, server time from the x-envoy-upstream-service-time header, p50, p95, max, and requests per second.
- Server p50 measured 75 to 90 ms versus TypeSafe's documented ~100 ms; sending all 100 at once pushed round-trip p50 to about 2 s and triggered 529s.
- There is deliberately no regex matching or provider verification, and the Noul question is pinned to jev-1.13.0 in questions.py.
Quickstart
bash
echo "TYPESAFE_API_KEY=YOUR_API_KEY" > .env
uv run python main.py
uv run python main.py configWatch out
No license file, so reuse terms are unclear. Needs Python with uv and a TYPESAFE_API_KEY; fixtures use randomly generated values that secret scanners may still flag, and the config and edge batches are smaller.
Questions about this entry
Short answers drawn from the linked project or page.
Purely Jev, by design. There is deliberately no regex matching and no provider verification, so the numbers score the model rather than a rule-based hybrid. That also means the comparison most people want, a real scanner such as Gitleaks on the same files, is not part of this run.
Every secret value is randomly produced with openssl and ssh-keygen and has never been valid, though the formats follow each provider's public token layout. So the shapes are right and nothing is live, and you should expect secret scanners and GitHub push protection to flag the fixture file anyway.
That the hard class is low-entropy and unprefixed. Those 20 config-shaped cases scored 15 of 20 and 16 of 20 across two runs at a 0.5 threshold with AUC 0.939 and 0.955, against 99 of 100 on the main set, and almost every miss was a non-secret pushed into the review band rather than a missed secret.
Not as written. The script fires 16 requests at a time and opens connections before timing starts, and sending all 100 at once pushed round-trip p50 to about 2 seconds and triggered 529 Overloaded retries, which the report counts. Budget for tuning concurrency and handling retries yourself.
Put TYPESAFE_API_KEY in .env and run `uv run python main.py`. The default is the 100-case set; add `edge` or `config` (for example `uv run python main.py config`) to run a smaller batch. The report prints accuracy, AUC, recall, precision, Brier score and latency percentiles.
By design, so it scores Jev itself. Each case is a single Noul question asking whether `content` contains a real secret credential someone could use, and the returned probability is compared with the expected label. Jev receives only file_path and content, never the neutral case names.
main.py sends 16 requests at a time and opens connections before timing starts. Sending all 100 at once pushed the round-trip p50 to about 2 seconds while server time stayed near 90 ms, and triggered 529 Overloaded retries, so the report counts SDK-retried requests.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Secret detection with Jev, tested
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
More Jev resources like this
A public benchmark comparing Jev with Claude Haiku 4.5 on whether an email agent should click a link, over 2,000 PhishNChips emails, with calibration, latency, cost, and five signal questions.
Jev Practices & Patterns#community#python#security
Community
anisselbd/jev-phishing-benchPython
anisselbd/jev-phishing-benchPythonUnofficial companion to dspy-typesafeify that records and replays TypeSafe calls to measure calibration, selective risk, confidence-gated abstention, latency, tokens and modeled cost for Jev decisions.
Jev Practices & Patterns#community#python#evaluation
Community
jmanhype/jev-dspy-labPython
jmanhype/jev-dspy-labPythonBlind security benchmarks for Jev on public corpora: 662 prompt-injection messages and 200 matched vulnerable-code pairs, with raw per-sample results and a TUI dashboard.
Jev Practices & Patterns#community#go#benchmarks
Community
Gaurav-Gosain/jev-sec-benchGo
Gaurav-Gosain/jev-sec-benchGoAn independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.
Jev Practices & Patterns#community#python#routing
Community
FirasSX914/JanusPython
FirasSX914/JanusPythonA ModernBERT-based open decision model that evaluates Choice, Score, and Noul schemas in one non-autoregressive pass; Verdict 2.0 reports 77.10% accuracy on 2,000 held-out enterprise decisions.
Jev Practices & Patterns#community#python#open-models
A self-hosted implementation of TypeSafe's Jev System One API powered by the 400M-parameter GLiFormer encoder: it serves choice, score, and noul and drops into the official typesafe-sdk via TYPESAFE_BASE_URL, but trails Jev on reasoning-heavy tasks.
Jev Practices & Patterns#community#python#open-models
Community
logan-markewich/jeffPython
logan-markewich/jeffPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.