Question 1

What question does the Jev detector answer in the bench?

Accepted Answer

One Choice with four options: same, fact_differs, action_differs, or specificity_differs. Anything other than same counts as changed, and probabilities come back with the answer so the threshold lives in your code.

Question 2

Why compare Jev to exact match and embeddings?

Accepted Answer

Text diffing is no help because re-sampled answers differ every time. Exact match caught all 19 changes but flagged 84 false alarms, and cosine embeddings under 0.75 caught only 7 of 19, so the bench measures false-alarm cost.

Question 3

How is ground truth derived when nobody labels the pairs?

Accepted Answer

The fixture is one support agent with a 9-line policy and 16 questions, then seven config changes, three cosmetic and four rule changes. Whether a pair changed is derived from what the rule mechanically does to the answer, and pairs where the generator ignored the rule are dropped, three of them in the shipped corpus. Two earlier hand-written lists of affected questions were wrong, which is why derivation replaced them.

Question 4

What do the run numbers look like across runs?

Accepted Answer

Jev caught 19 of 19 with 13 false alarms at p50 0.41 s and $0.0028 per run; the LLM judge 18 of 19 with 19 false alarms at 1.60 s and $0.0071; the three-tier stack 19 of 19 with 33 false alarms at 0.00 s and $0.0024. Across runs Jev moves between 12 and 13 false alarms and the judge between 18-19 caught and 19-20 false alarms.

Question 5

How does a config change that does nothing behave?

Accepted Answer

The rule always recommend the Team plan changed nothing, because the agent already recommended Team every time, so ground truth is zero. The three-tier stack flagged 8 of 16 answers anyway, eight pointless reviews from a no-op edit, while Jev flagged 4. That row is the clearest case for caring about false alarms over recall.

Question 6

How do I run it, and what caveats does it state?

Accepted Answer

npm install, copy .env.example to .env with OPENAI_API_KEY and TYPESAFE_API_KEY, then npm run bench; corpus.json ships so it reproduces without paying to regenerate answers, and npm run generate rebuilds it with 128 gpt-4o-mini calls at temperature 0.7. It is MIT licensed, and it says plainly that it is a fixture with one agent, one domain and 16 questions, on a single run per detector.

Question 7

Why not just diff the answers textually?

Accepted Answer

Because re-sampling changes the words every time regardless of meaning. The bench re-samples one config with no rule change and the ground truth is 0, yet exact match reported false alarms on reworded and re-sampled pairs because string equality cannot tell a paraphrase from a policy change. That is the whole reason the fixture ships pairs rather than diffs.

Jev Regress Bench

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community