User repository on GitHub
Jev (TypeSafe) vs Claude Haiku 4.5 on 2 000 phishing emails: accuracy, calibration, latency, cost. Reproducible benchmark.
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Practices & Patterns
Jev vs LLM: a phishing decision benchmark with a calibration audit
A public benchmark comparing Jev with Claude Haiku 4.5 on whether an email agent should click a link, over 2,000 PhishNChips emails, with calibration, latency, cost, and five signal questions.
- Stars
- 9
- Language
- Python
- Reading time
- 4 min
- Published by
- Community
- Author
- anisselbd
- Last verified
- Sep 2026
About this project
Public, reproducible comparison of Jev (TypeSafe AI's System One model, launched 15 September 2026) against a classic LLM on one security decision: should an email agent click the link in this email? The question this repo answers with numbers: is Jev accurate enough, are its probabilities calibrated, and how much faster and cheaper is it than an LLM on the same 2 000 emails?
Quoted from the project's README. Read the full README.
Works forJev for Evaluation
Used inJev for Security & Privacy
Highlights
- Both systems saw the same 2,000 PhishNChips v5.2 emails in seeded order, one call per email, no concurrency.
- Jev was 62.6% accurate with AUROC 0.689 and ECE 0.154; Claude Haiku 4.5 was 81.3% accurate with AUROC 0.837.
- Jev won on cost and speed: $0.038 per 1,000 emails at list price and 239 ms p50 from France versus $0.462 and 687 ms for Haiku.
- A cross-validated logistic regression on Jev's five signals reached 95.0% on held-out half B, against 91.8% for a two-feature regex regression.
- The signal result was re-controlled after review: Jev's best single signal lost to both the regex (89.4% vs 91.8%) and Haiku asked the same question (94.2%).
Quickstart
bash
cp .env.example .env # fill in the TypeSafe and Anthropic keys
uv run run_jev.py --limit 10 # smoke test, prints raw answers
uv run run_llm.py --limit 10Watch out
No license file, so reuse terms are unclear. Needs Python 3.13, uv, and both TypeSafe and Anthropic keys; published recall on the dataset varies with the system prompt, and one pass per arm limits stability claims.
Questions about this entry
Short answers drawn from the linked project or page.
No, and the repo says so after adding the control it was originally missing. On held-out half B, Jev's best single signal reaches 89.4% against 91.8% for a list of shorteners, free hosts and document-sharing hosts plus an eTLD+1 comparison, McNemar p = 0.0032. What Jev keeps is the decomposition's price: roughly 27 times cheaper and 5 times faster.
One prompt per system, and the dataset shows how much that matters: published recall across its own grid runs from 27% to 96% with the system prompt alone. Jev's sensitivity was probed with three alternative wordings in the same call, where the best gained 0.6 accuracy points and the worst lost 4, so no rewording changes the conclusion.
Not clearly. Haiku's best single signal reaches 94.2% against Jev's 89.4%, p below 0.0001. The regression on Jev's five signals reaches 95.0% against 93.2% for Haiku's, but that gap is not significant at McNemar p = 0.063 and Haiku's AUROC is higher, 0.991 versus 0.982, for 25 times the cost.
Reproducible: the uv scripts download the release, verify SHA-256 values from the manifest, append to results/raw jsonl and skip emails already answered so a resumed run does not pay twice, and every proportion carries a Wilson interval. Not real mail: bodies are LLM-generated by the dataset authors and ground truth comes from URL reputation feeds rather than a human reading each email.
One request per email with the email object as state and nine typed questions in it: a verdict choice, a mirror noul for the same question, five signal nouls covering domain mismatch, free hosting, lure, urgency and generic sender, plus two alternative verdict wordings. The README publishes the exact text and fixes the headline wording before any answer was read.
Jev flipped 2.2% of labels on a second pass, with a mean absolute difference of 0.017 and a maximum of 0.15, and 1.0% on a 200-email subset twelve hours later. Haiku flipped 0.7% on 300 emails with 98% identical and a maximum difference of 0.65, so its agreement looks tighter while its swings are far larger.
Python 3.13 and uv, plus a TypeSafe key and an Anthropic key in .env. prepare_data.py downloads the dataset and checksums it, net_floor.py measures the network round-trip floor to each API host so latency numbers can be read honestly, run_jev.py --limit 10 is the smoke test, and LLM_RPM paces the Haiku arm for free-tier rate limits.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
Blind security benchmarks for Jev on public corpora: 662 prompt-injection messages and 200 matched vulnerable-code pairs, with raw per-sample results and a TUI dashboard.
Jev Practices & Patterns#community#go#benchmarks
Community
Gaurav-Gosain/jev-sec-benchGo
Gaurav-Gosain/jev-sec-benchGoA measurement harness that sends each file snippet to Jev as one Noul question asking whether the content contains a usable secret credential, then scores the returned probability against the expected label.
Jev Practices & Patterns#community#python#security
Community
teyhouse/jev-secret-detectionPython
teyhouse/jev-secret-detectionPythonA ModernBERT-based open decision model that evaluates Choice, Score, and Noul schemas in one non-autoregressive pass; Verdict 2.0 reports 77.10% accuracy on 2,000 held-out enterprise decisions.
Jev Practices & Patterns#community#python#open-models
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Jev Practices & Patterns#community#python#benchmarks
Community
fstandhartinger/jevbenchPython
fstandhartinger/jevbenchPythonAn experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Jev Practices & Patterns#community#python#evaluation
Community
danielgshea/jev-as-a-judgePython
danielgshea/jev-as-a-judgePythonA probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.