Question 1

How can jev-verify check outputs without calling the Jev API?

Accepted Answer

Jev's confidence is a deterministic function of the probabilities it returns: C = (N * p_max - 1) / (N - 1), established by Stanislav Yurin over more than a million live tests. jev-verify recomputes that identity and the expected-score formula from published JSON artifacts, so an answer that does not obey them did not follow the same law.

Question 2

What did jev-verify's ecosystem census find?

Accepted Answer

Across 275,852 categorical answers found in 1,054 repositories, 98.22 percent conformed to the identities. The deviations split into repositories using a different confidence law, rounding-level differences at publishing precision, and fabrication flags that all landed in hand-authored fixtures rather than vendor-channel or recorded API responses.

Question 3

What is the fabrication flag, and where did it land?

Accepted Answer

In several fixtures the confidence equals the fractional part of the score, so 2.58 gives 0.58 and 2.66 gives 0.66. A calibration statistic cannot track a decimal remainder, so it is a generator artifact. The flags appear in ax-llm/ax and 0xPlaygrounds/rig's rounded.json, and in zero vendor-channel or recorded-response samples, which the identity test flags independently.

Question 4

Why treat published artifacts as separate populations?

Accepted Answer

Conflating them produces a false number. Vendor-channel catalog examples conform 10 of 10 with zero flags, recorded API responses 843 of 854, reimplementations named after Jev 19 of 80, and hand-authored fixtures 115 of 296 with all 121 fabrication flags. An earlier census over-reported by scoring every probabilities map for summing to 1, misreading multi-label outputs as fabrication.

Question 5

What are the limits of a deviation finding?

Accepted Answer

A deviation means the output does not obey the identities real outputs obey, not proven fabrication, and innocent explanations should be checked first: deliberate rounding, an older model version, or a reimplementation. It covers Choice and Score only, since a Noul carries no confidence to check, makes no live API call, and every number comes from published artifacts.

Question 6

How much of this is the authors' own discovery?

Accepted Answer

Not much, and they say so. They derived the same identity independently from 843 published artifacts with no API access before finding Yurin's and primeline's work, with residuals of mean 0.005 and max 0.0158 sitting inside Yurin's bounds over 738,164 live answers. primeline.cc independently replicated it on 1,203 and 801 answers and added monotonicity. MIT licensed, with 30 known-answer tests.

Question 7

How is the census scoped, and which answers are skipped?

Accepted Answer

The identities are defined only on a single categorical distribution over mutually exclusive outcomes, so the checker first classifies each answer's task structure and scores only categorical answers. Multi-label outputs, detected by a multi-hot truth sibling or by the fact that a real distribution cannot carry two probabilities each above 0.5, and batched containers with batch_size above 1 are skipped, never scored. Of 1,061 repositories in scope, 1,054 were scanned and 88 contained Jev answers.

Jev Verify

About this project

Highlights

Questions about this entry

Reactions & coverage

Related terms

More Jev resources like this

From the Jev community