User repository on GitHub
Zero-shot spam filtering with TypeSafe Jev Noul questions, compared with TF-IDF baselines
- Forks
- 1
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Practices & Patterns
Zero-shot email classification with TypeSafe's Jev
An exploratory study of zero-shot ham, spam, and phishing classification with Jev: enriched email context raised main-set accuracy from 93.62% to 98.64%, within 0.23 points of a trained TF-IDF baseline.
- Stars
- 3
- Language
- Jupyter Notebook
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- bitnovus
- Last verified
- Sep 2026
About this project
TypeSafe’s Jev reached 98.64% accuracy on a 5,733-email ham/spam/phishing test using written category definitions and email context, without task-specific fine-tuning or labeled examples in its requests. A TF-IDF logistic regression classifier trained on roughly 4,600 labeled messages per fold reached 98.87% with the same evidence.
Quoted from the project's README. Read the full README.
Used inJev for Software & DevTools
Highlights
- On 9,886 messages, adding link destinations, Reply-To, and attachment metadata raised accuracy from 93.62% to 97.98% with the question unchanged.
- Phishing recall rose from 85.71% to 98.43%, while legitimate messages flagged as phishing stayed at one.
- With evidence-focused wording the main set reached 98.64%, 0.23 points behind a TF-IDF logistic regression trained on about 4,600 labeled messages per fold.
- On 853 phishing messages from 2024-25, enriched Jev caught 95.31% versus 75.26% for regression trained on older mail.
Quickstart
bash
export TYPESAFE_API_KEY=YOUR_API_KEY
uv run python experiments/jev-context/evaluate.py --limit 3Watch out
MIT-licensed. Results are exploratory on jev-1.13.0 and public corpora that may have leaked into pretraining, and dataset licenses are separate from the code license.
Questions about this entry
Short answers drawn from the linked project or page.
That cannot be ruled out, and the README says so, calling the results exploratory on one pinned model version with no way to establish whether the corpora were seen in pretraining. A second caveat matters more: specifications and experiment choices were informed by earlier labeled errors, so it is zero-shot inference, not development free of labeled feedback.
On the main set it comes within 0.23 points, 98.64% against 98.87% for enriched TF-IDF regression trained on roughly 4,600 labeled messages per fold. On the fresh set regression keeps the lead at 96.39% versus 95.76%, including after removing 902 near-copy messages, so the answer is not everywhere.
The input, by a wide margin. Holding the question fixed and adding link destinations, Reply-To, and attachment metadata moved main accuracy from 93.62% to 97.98% and phishing recall from 85.71% to 98.43%. Changing the wording afterward added only 0.66 points and caught fewer recent phishing messages.
On the main set, legitimate mail flagged as phishing stayed at 1 through the enrichment change and spam misfiled as phishing fell from 16 to 15. Be careful with the later phishing-only set, though: it contains no legitimate mail at all, so its recall figure says nothing about false positives.
It kept the original subject, sender and a 6,000-character text body, then added Reply-To, visible link text paired with its actual destination and parsed hostname, and attachment filenames and content types. URLs are extracted from the textual MIME parts and no links are visited.
Keeping the question and category definitions unchanged, adding the extra fields raised main accuracy from 93.62% to 97.98% and phishing recall from 85.71% to 98.43%, while legitimate messages flagged as phishing stayed at one. A further wording change brought accuracy to 98.64%.
A TF-IDF logistic regression trained on roughly 4,600 labeled messages per fold reached 98.87% on the main test with the same evidence, 0.23 points above enriched Jev. On 853 phishing messages from 2024-25, enriched Jev caught 95.31% versus 75.26% for regression trained on older mail.
More Jev resources like this
A self-hosted implementation of TypeSafe's Jev System One API powered by the 400M-parameter GLiFormer encoder: it serves choice, score, and noul and drops into the official typesafe-sdk via TYPESAFE_BASE_URL, but trails Jev on reasoning-heavy tasks.
Jev Practices & Patterns#community#python#open-models
Community
logan-markewich/jeffPython
logan-markewich/jeffPythonIndependent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Jev Practices & Patterns#community#python#benchmarks
Community
fstandhartinger/jevbenchPython
fstandhartinger/jevbenchPythonAn experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Jev Practices & Patterns#community#python#evaluation
Community
danielgshea/jev-as-a-judgePython
danielgshea/jev-as-a-judgePythonA probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonAn open, local-first alternative to Jev that reproduces the Choice, Score, and Noul interface on commodity zero-shot NLI models with temperature scaling and conformal abstention, and runs offline with no API key.
Jev Practices & Patterns#community#python#calibration
Community
rupeshpoojary9/poorjevPython
rupeshpoojary9/poorjevPythonA benchmark that runs Jev on all 6,257 text traces of Who&When Pro to attribute agent failures, scoring it with the official pinned scorer against the paper's GPT-5.4, Claude Sonnet 4.6, GLM-5, and Qwen3.5-122B baselines.
Jev Practices & Patterns#community#python#benchmarks
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.