Question 1

Are the published numbers a real three-model run, or something that could have been edited?

Accepted Answer

All three evaluators are recorded under one run ID with the same corpus hash and complete coverage of all 100 cases. Artifacts are exclusive-write and reject credentials, absolute local paths, duplicates, missing or unknown cases and invalid hashes, and the dashboard build refuses a superseded corpus hash. The README also states that the earlier v2 to v4 runs were published before those checks and explains why each was dropped.

Question 2

Does Jev actually win here, or is this just a demo harness?

Accepted Answer

The README calls it a descriptive, synthetic demo and stops there. Scoring leads with balanced accuracy, the unweighted mean of per-class recall, and reports recall with the vulnerability family required, family accuracy, precision and false-safe rate alongside it. Explicit errors stay incorrect and remain in every denominator, and Jev's own distribution is scored with a multiclass Brier score rather than compared to a frontier model's confidence.

Question 3

What does it cost to reproduce the recorded run myself?

Accepted Answer

The README puts a full three-model recording at about 3.3 hours and roughly $40, and it needs macOS, Codex CLI 0.147.0, Claude Code 2.1.278 and a TYPESAFE_API_KEY. Cheaper paths exist: npm run dashboard:fixture and npm run demo exercise the whole pipeline with explicitly simulated adapters and no paid calls, and npm run check runs the tests and dashboard builds, which never make model calls either.

Question 4

How do I know the corpus is not leaking the answer to the model?

Accepted Answer

Span paths are kept label-neutral, and the private ledger.contextResolution metadata that records how ambiguous context was resolved is never sent to any evaluator. A loader check enforces this on every case, which is the specific defect that invalidated earlier runs, where dispositions leaked through file paths and broken-access-control cases could not be decided at all.

Question 5

What is actually in the 100-case corpus, and who evaluates it?

Accepted Answer

Three families split across three outcomes: injection 34 cases, broken access control 33 and SSRF 33, totalling 34 vulnerable, 33 safe and 33 insufficient context. Jev via the TypeSafe SDK, OpenAI's Terra and Anthropic's Opus each answer every case, five passes each for 1,500 decisions.

Question 6

What stops the builder from publishing a broken or one-sided run?

Accepted Answer

A degenerate run, where one evaluator returned a single outcome for every case, is refused outright; --allow-degenerate publishes it only with a visible warning. Artifacts are exclusive-write and reject credentials, absolute paths, duplicates, missing or unknown cases and invalid hashes.

Question 7

Do the tests or the local preview make paid model calls?

Accepted Answer

No. npm run dashboard:fixture and npm run dashboard:preview exercise the complete dashboard path with explicitly synthetic fixture results, and npm run check runs the tests and dashboard builds; neither makes paid model calls, so no credentials are needed.

System One Security Triage

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community