Docs
CommunityJev Benchmarks & Evaluations
Jev-class CVE Bench
A 30-option CWE weakness-class benchmark for Jev-class decision models: pick the class of a real NVD vulnerability description, with a 150-case dev split scored locally and 1,500 held-out test cases on the leaderboard.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- trapstreet
- Last verified
- Oct 2026
- Languages
- Shell
- Capabilities
- Open source
Highlights
- 30 CWE options per case; dev holds 150 cases with answers, 5 per class, and test holds 1,500 held-out cases, 50 per class.
- A no-model string matcher scores 0.478 because 392 of 1,500 descriptions contain their own class name; random guessing scores 0.033.
- Confidence is optional and never scored; overconfidence is reported as mean stated confidence minus accuracy.
- Jev 1.13.0 and open replicas sit on the decision-layer-bench leaderboard; the project states it is not affiliated with TypeSafe.
- Cases rebuild monthly from CVEs published 2026-07-01 to 2026-09-15; each refresh adds a config and never changes an old one.
Quickstart
bash
uv tool install trap-cli
tp run <your-solution> --task cve-weakness-class
tp auth login
tp submit <your-solution> --task cve-weakness-classWatch out
Card reviewed 2026-10-17; the card warns that descriptions are NVD's exact text so anyone can look up the answer, that the published dev answers make dev scores noisy (margin about ±0.036) and cannot rank models, and that test answers are withheld. NIST and MITRE do not endorse the benchmark.
Questions about this entry
Short answers drawn from the linked project or page.
One task in decision-layer-bench: read the published description of one software vulnerability and pick the weakness class it belongs to from 30 CWE identifiers, as a Jev-class typed decision at 30 options.
Dev has 150 cases with answers included (5 per class) for scoring on your own machine; test has 1,500 cases (50 per class) with answers withheld for the leaderboard. No case appears in both splits.
A plain string matcher with no model scores 0.478 because 392 of the 1,500 descriptions name their own class; random guessing scores 0.033, so 0.478 is the documented floor.
Nothing for the score: a confident wrong answer and an unsure wrong answer both score zero. Confidence is reported separately as overconfidence, the average stated confidence minus accuracy, and negative values mean under-confidence.
Install trap-cli and run the task locally with tp run before authenticating; submissions go to the public leaderboard at trapstreet.run, where every entry is scored by the same included judge on the same cases.
Recorded as mit-construction-public-domain-data: the case construction code is MIT and the vulnerability records come from the public-domain National Vulnerability Database, while the CWE classes come from MITRE's free catalogue version 4.20.
Sometimes: the descriptions are NVD's exact text, so a search engine can find the original record with its answer, and descriptions mentioning a CWE or CVE identifier are removed from the pool for that reason but cannot be fully filtered out.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
Harmful/benign classification and answer-choice robustness for five type-safe models: 83,183 source requests from XSTest, OR-Bench and HarmBench in original, reversed and order-only variants.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Teacher-labelled distillation pack for the one-token Qwen decision track: 10,044 rows across train, eval and calibration JSONL files, plus 1,708 labelled fixtures. Labels come from jev-1.13.0, and the card says not to train on the eval or calib splits. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
An empirical comparison of Jev as a typed judge for retrospective agent-trace security against four generative judges on 5,219 trajectories. Jev averages positive-class F1 77.8 versus 74.1 for GLM-5.2, leading on two of four benchmark collections.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Raw traces from retired, versioned native ToolEmu and AgentDojo runs, including failures and partial trajectories, preserved as original bytes with SHA256 manifests and upload revisions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.