Question 1

What is the jev-test benchmark?

Accepted Answer

A pre-registered benchmark from clduab11. A small model on a laptop writes the answers, while TypeSafe Jev (jev-1.13.0) makes every call the small model is bad at: whether to search, which pages count, whether there is enough evidence, and whether each sentence is backed by what it cites.

Question 2

How does the judged pipeline work?

Accepted Answer

SearXNG finds the pages and MemPalace remembers them; Gemma 4 E2B writes the answers; Jev judges the passages. The rules were fixed in docs/JUDGMENT_SPEC.md before the first run, and the pipeline and the 20-question pilot are documented in docs/DESIGN.md.

Question 3

What did Jev's confidence measure well?

Accepted Answer

Across 9,075 passages from 349 questions, Jev's answer to does this passage hold the answer? had AUROC 0.899, with a 95% interval of 0.881 to 0.916, against 0.5 for random scores. With golds longer than 7 characters it was 0.902, and measured within each question 0.925.

Question 4

Did the judged pipeline beat the baseline end to end?

Accepted Answer

No. It scored 0.612 against the no-judge baseline's 0.740, missed its main pre-registered bar, and made about the same number of mistakes on the questions both answered. It lost because it declined far more questions, as written up under Open problems.

Question 5

How did Jev compare with Gemma judging itself?

Accepted Answer

On a 20-question pilot over 473 passages from 12 questions, Jev's AUROC was 0.926 against Gemma's 0.804. Gemma marked 189 passages 1.00; only 40% (76) held the gold answer, and inside that fully-certain group Jev still separated the two kinds at AUROC 0.835.

Question 6

What does Gemma's verbalized confidence look like?

Accepted Answer

On 3,885 judgments both models made, they gave the same yes-or-no call at 0.5 on 82.8%. Gemma used 12 distinct values and put 99.6% of answers on three of them (0.00, 1.00, 0.50); Jev used 99 distinct values and placed 30.3% of answers between 0.10 and 0.90, against Gemma's 6.3%.

Question 7

What ships in the repository for reuse?

Accepted Answer

harness/audit.py checks any judge's confidence scores, and harness/grading/recall.py is the model-free check that marks a passage positive when the gold answer string appears in its text (golds of 4 characters or fewer are dropped). The repo is Python, MIT licensed, and includes the judgment spec and design doc.

jev-test (Gemma 4 E2B judged by Jev)

Highlights

Questions about this entry

From the Jev guides

From the Jev AI model community