Question 1

Can I run it in my CI pipeline?

Accepted Answer

Yes. A GitHub Action runs the suite against your URL and fails unless the verdict is conformant, writing the report to the job summary; the mock also stands in for Jev with no key or network.

Question 2

Why does confidence matter so much here?

Accepted Answer

A tuned threshold silently changes its automation rate on another server because confidence is computed at least five ways; the spec pins the documented formula and the proxy recomputes it.

Question 3

What exactly did the eight-server survey find?

Accepted Answer

Measured 2026-09-24 at pinned commits, only kev at 5.8k stars and decider at 338 passed every MUST requirement. Four of the eight cannot take the documented 255 choice options, stopping at 26, 50, 64 or between 64 and 128, and five compute confidence differently from the documented formula: 1 minus normalised entropy, the top probability, the top-two margin, or the score spread over L minus 1.

Question 4

How much does the confidence formula change an automation rate?

Accepted Answer

Applied to the same 204 recorded answers at a 0.9 threshold, TypeSafe's formula auto-accepts 58% of them, the top-probability definitions accept 63%, the top-two margin 57%, and Laya's 1 minus normalised entropy only 30%. Laya's definition rejects 56 of the 118 answers TypeSafe's formula accepts, and decider's accepts 10 that TypeSafe would send for review.

Question 5

What does the proxy actually fix, and what does it refuse?

Accepted Answer

The normalising proxy recomputes choice, score, legend and confidence and fixes 38 of the 47 injected faults. It refuses 9 with a 502 rather than passing them through, so a caller gets a loud failure instead of a quietly wrong answer. A server can be fully conformant and still give bad answers, since the checks judge the wire contract rather than accuracy or calibration.

Question 6

How do I know the suite can actually catch things?

Accepted Answer

The mock implements the spec exactly and can break any requirement on purpose, shipping 47 deliberate faults, and every failure was checked by hand against the recorded exchange. CI runs each check against the clean mock, which must pass, against a noisy mock over many seeds, which must never fail, and against each fault, where the targeted requirement must fail and one fault must not smear failures into unrelated MUSTs.

Question 7

What does a failing check actually show me?

Accepted Answer

For each failure the report names the requirement, the case and the raw exchange. The example given is a Score answer where the server returned 0.9973 while its own probabilities give a sum of i times p of 0.8640, because the score was computed before the probabilities were tempered, taken from the jeff report. The verdict is one of conformant, not conformant, not tested or incomplete, with distinct exit codes.

Jevcompat

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community