CVE triage is a risk-triage problem that looks like one decision and is really three. One question assigns the weakness class. Another estimates exploitability in your environment. A third decides where the item sits in the patch queue. A single "is this critical" score collapses information you need, and a queue built on it will be wrong in ways you cannot explain.
Separate the questions
Severity is usually given: CVSS, the vendor advisory, and asset inventory supply most of it. What is missing is context: whether the service is internet-facing, whether exploitation needs authentication or user interaction, whether the vulnerable path is reachable, and whether a patch will break something.
Those are narrow and answerable. "Should we patch this first" is not: it mixes exploitability with business criticality, change windows, and outage cost. The model supplies evidence; the queue policy belongs in code.
Benchmarks and their floors
The Jev-class CVE Bench is the most direct benchmark for the classification step. It gives a model 30 CWE options and asks for the weakness class of a real NVD description. Dev has 150 labeled cases, five per class; test has 1,500 held-out cases, 50 per class.
The floor matters more than the headline. A no-model string matcher scores 0.478 because 392 of the 1,500 descriptions name their own class; random guessing scores 0.033. Anything below 0.478 is worse than reading the text. Dev's margin is about ±0.036, so it cannot rank models. Confidence is optional and never scored; overconfidence is reported as mean stated confidence minus accuracy.
Jev-IDS sits at the other end of the pipeline. It serializes one network flow per request and asks two questions: a binary attack probability and a finite-choice traffic category. On a 300-flow NSL-KDD pilot split, across 5,400 decisions, it reports F1 0.859, precision 0.941, recall 0.790, and novel-attack recall 0.838. It claims 4.8x faster and 3.8x cheaper operation than GPT-5.6 Luna, with 1.5x higher novel-attack recall, and 15x fewer false alarms than a low-data Random Forest. Increasing k to 2 reduced F1 to 0.839. The caveat is decisive: a small, dated benchmark with no named repository does not establish modern performance.
Triage comparisons
Jev Triage is a Spring Boot benchmark on warehouse goods-receipt remarks. The dataset is 200 synthetic remarks, 113 German and 82 English plus 5 mixed, split 50/50 into calibration and test. On the 100-remark test split, run three times, category accuracy was 41.0% for Java keyword rules and 100% for Jev 1.13.0 and Claude Haiku 5.5. Jev reached 72.0% block accuracy against 66.0% for rules, and 73.3% severity against 48.0%.
Rules are good at the binary block decision and bad at category and severity. But the dataset is small and synthetic, the criteria and hazmat rule were designed after first-run errors, and a few points are noise.
system-one-security-triage is more careful about provenance. It runs 100 synthetic TypeScript/Node cases across injection, broken access control, and SSRF: 34 vulnerable, 33 safe, 33 insufficient context. Five passes per case produce 1,500 decisions. Jev answers one five-way Choice, and its full distribution is scored with a multiclass Brier score alongside OpenAI's Terra and Anthropic's Opus. The README calls it a descriptive demo, not a powered benchmark, and puts reproduction at about 3.3 hours and $40. Three earlier runs are superseded because the corpus leaked labels through file paths or left cases undecidable.
Gates, not oracles
The most credible production report comes from Vega. It put Jev in front of an alert triage agent on two tenants. As a gate, it closed 15% of alerts on the busy tenant and a third on the quiet one, about 98% right by three LLM judges whose labels Vega's analysts verified by hand.
Vega is explicit that Jev could not decide alerts outright; the tenant's own history made the gate work. The post lists the limits plainly: Jev cannot count, compares numbers badly, is confused by negations, reads literally, and tops out around 32k tokens of state. The analysis is vendor-authored, on two of Vega's tenants, with labels verified by its analysts. Treat it as an existence proof, not an independent evaluation.
That is the right role for a decision model in CVE work: close the obvious cases, flag the ambiguous ones, and leave the actual queue policy to code.
Calibration and queue economics
A patch queue is a capacity constraint: you review a fixed number of items per day, so the useful operating point is precision at that capacity, not corpus-wide accuracy.
TypeSafe's confidence documentation explains the mechanics. Confidence is a 0-1 statistic derived from the distribution, not a separate probability; Noul answers carry none. The docs approximate three-option confidence as (3 × largest probability - 1) / 2 and route below 0.5 to a human, requiring above 0.9 to auto-approve a transfer. The full distribution still comes back.
For triage, the asymmetry is the point. A false "not exploitable" can leave a critical service unpatched. A false "exploitable" costs an hour of analyst time. Set the threshold for the first error, then measure how much review capacity the second error consumes.
COGNIT-Guard is not a Jev evaluation, but it is a useful architecture reference. It pairs a calibrated CPU gatekeeper with confidence-gated escalation to a 322M model on an NPU. On a 607-item unseen split it reports 98.85% accuracy, 0.42% benign false-positive rate, and 1.12% ECE; NPU inference runs at 21.77 ms and the cascade averages 41.63 ms. Out-of-domain accuracy fell from 60.33% to 56.81%, and replay restored it to 64.10-65.05%. The numbers are self-reported and tied to Huawei Ascend hardware. The transferable idea is the split: cheap local gate first, model escalation second.
An architecture that fits
| Stage | Where it runs | What it answers |
|---|---|---|
| Enrichment | Code | CVSS, EPSS, asset criticality, exposure, reachability |
| Weakness class | Decision model | Which CWE class, from a fixed list |
| Exploitability | Decision model | Does the description state remote, unauthenticated, user interaction, public exploit |
| Queue policy | Code | Priority from severity, exposure, and business weight |
| Remediation | Agent with a gate | Apply the patch, run tests, open the change |
Do not ask the model to count, compare dates, or infer an unstated condition. The jaggedness notes list those edges explicitly: counting is unreliable and error grows with size, dates are read as text, and structural invariants do not hold. Extract components with Choice and do the arithmetic in code.
JEV Attack Surface Analysis is a useful upstream tool for the reachability question. It ranks likely vulnerabilities for about a cent per repo, measured at $0.010 for PyGoat and $0.007 for NodeGoat. It ranks candidates and does not prove a vulnerability, which is what a triage input should do.
The action after triage also needs a gate. jev-guard checks shell commands, file writes, and file reads before a coding agent runs them and returns ALLOW, ASK, or DENY, with a local fast path and a fail-safe ask when the API is unreachable. Patching is an irreversible action; the triage decision should not be the only thing standing between an agent and production.
Test before queueing
Build a historical set with known outcomes, including the CVEs you missed. Measure detection at review capacity, false-safe rate separately, and confidence calibration. Re-run when the model version, question text, or option order changes. The question-registry article covers the regression mechanics, and the jaggedness article covers the model-level edges to put in the suite.
A decision model is a good triage clerk. It is a bad queue owner. Keep those jobs separate and the system stays explainable when an incident review asks why a CVE waited.
