Q: How do I replay a pack offline?

Run uvx jevassert check packs/ -p packs/ /predictions.jsonl, which replays the committed evidence deterministically and for free; to record fresh predictions against any Jev-compatible endpoint use uvx jevassert record, with --base-url for a local replica.

Question 1

How do I know these numbers are not marketing?

Accepted Answer

The recordings are committed and the checks cost nothing. Each backend has a directory under `results/` with its raw recording, per-pack accuracy, ECE, cost and latency, and the exact command to reproduce the column. `uvx jevassert check <pack> -p predictions.jsonl` replays that committed evidence offline, deterministically and for free, and CI fails if the generated scoreboard goes stale. The README's own framing is that vendor evals are vendor-run and this runs the other direction.

Question 2

Does Jev actually beat Claude here?

Accepted Answer

On two of the packs, and the repo does not hide the rest. Jev leads on citation-support at 0.979 against 0.941 and on rag-passage-relevance at 0.921 against 0.899, while Sonnet leads on support-triage at 0.913 against 0.898 and six packs are statistical ties. Where the gap is consistent it is calibration and cost, with better calibration on 8 of 9 packs and roughly 250 times cheaper per case, while a local 7B open-weights model trails well behind on accuracy.

Question 3

What is the `unknown` label rule, and why is it mandatory?

Accepted Answer

Because Jev cannot abstain, so every Choice and Score in every pack must offer an `unknown` option, the spec enforces it and CI checks it. Without that rule a forced answer on an undecidable item is indistinguishable from a real prediction, which is how most golden sets quietly go wrong. The README also gives the motivation for a registry at all: cookbooks and MCP servers ship their criteria once and never measure them.

Question 4

Are these packs usable without the runner, or is this lock-in?

Accepted Answer

Packs are plain YAML and JSONL, and the README states any parser can load one, with `scripts/validate.py` checking structure in CI. The canonical loader is `jevassert.packs`, which also records and replays, and you can point it at a local Jev-compatible replica with `--base-url`. There is deliberately no hosted registry or API and no CLI belonging to this repository. Raw dataset content is not committed, so derived packs ship their build script instead.

Question 5

How do I replay a pack offline?

Accepted Answer

Run uvx jevassert check packs/ -p packs//predictions.jsonl, which replays the committed evidence deterministically and for free; to record fresh predictions against any Jev-compatible endpoint use uvx jevassert record, with --base-url for a local replica.

Question 6

What gates does each pack ship?

Accepted Answer

A gates.yaml derived from committed evidence: min_accuracy is the baseline's 95% CI lower bound minus 1pp, max_ece is baseline ECE plus 3pp (never below 0.05), max_cost_per_case_usd is baseline times two, and max_p95_latency_ms is baseline p95 times 1.5.

Question 7

What is the disagreement harvest?

Accepted Answer

A review flywheel: five criteria passes across the packs removed systematic gold noise, cutting both-wrong cases from 55 to 18 and gaining Jev 0.4 to 6.0 percentage points on those packs.

Jev Packs

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community