Question 1

Jev alone shows a 0.000 false positive rate on that sample. How is that even possible?

Accepted Answer

It is the 0.5 threshold on a sample of 502 benign skills, and the README is upfront that scores vary a little between identical requests. More importantly the headline recall at that point is 0.755, not the 0.944 the cascade reaches. The held-out tuned threshold at 0.09 gives 0.906 recall at a 0.040 false positive rate, which is the more honest single-number read.

Question 2

Are these recall numbers measured, or is the cheap arm carrying the result?

Accepted Answer

They are measured on a 1,000-skill sample stratified by label, split and vector, with bootstrap intervals reported, and the protocol requires raw per-sample records including model version, token counts and truncation flag. But the band result depends on the LLM: Gemini caught 71 of 100 in-band malicious skills and Opus caught 94, both flagging 4 of 71 in-band benign. The cheap cascade matches the tuned threshold's recall at a fifth of its false positive rate.

Question 3

What does it cost to run the whole benchmark, and what will it run my credits on?

Accepted Answer

Jev bills input at $42 per billion tokens with free output, and the 7,944 states total 23.9M tokens, so a full run is about $1.00 by their arithmetic. Pin the model, measure 100 skills with --ids-from, then extrapolate rather than trusting the projection. Three transports exist: the SDK path, the Vercel gateway, and OpenRouter, which needs purchased credits — --backend auto prefers them in that order.

Question 4

Are the malicious skills actually executed anywhere in this repo?

Accepted Answer

No. The README is explicit that skills are inert text here and nothing in the repo executes them; fetch sparse-clones only the Dataset/Skills directory. That said, you are pulling roughly 1.5 GB of adversarial content onto your disk, so treat the checkout itself as untrusted input and keep it out of anything with credentials or network reach.

Question 5

What exactly is the state the model judges?

Accepted Answer

One request per skill, with the whole package as structured JSON: skill.name plus a files array where each entry has a path and content, so the model reasons about how the SKILL.md instructions and the bundled scripts interact rather than looking at either alone. Four typed questions come back: maliciousness as Noul, attack vector and behavior as Choice, and harm severity as a four-level Score.

Question 6

How is the cascade band decided?

Accepted Answer

Jev scores every skill; at or above 0.5 it is flagged, below 0.05 it passes, and only the band between goes to an LLM. On the 1,000-skill stratified sample that band is 171 skills, 17.1%, and the rule for whether the cascade was worth building was frozen before the run.

Question 7

What is in the corpus and where does it come from?

Accepted Answer

It uses the lxyeternal MalSkillBench corpus, about 1.5 GB and academic-research-only: 3,214 generated, 703 wild, and 27 tool-test malicious skills plus 4,000 benign, with labels taken from the benchmark's own source inventory. A second dataset with a nearly identical name exists but is explicitly a candidate second protocol, not the current target.

Jev Skillbench

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community