A benchmark number is a claim about a measurement, not a property of a model. It includes who ran it, on what data, against which baseline, with what scoring rule, and under what selection pressure. When a piece is missing, the number can still be true and still mislead.
Decision-model benchmarks now show all four at once: contamination, selection, weak construction, and reporting that omits what is needed to reproduce. The response is a checklist that separates measurement from marketing.
Four ways a benchmark stops measuring
Contamination is the obvious one: the test data, or a near copy, appeared in training. Selection is subtler: try many settings and keep the best public score, and the score now describes the selection process rather than the model. Construction is the label problem: if the gold answers come from a model, the ceiling is that model's agreement, not correctness. Reporting is the reproducibility problem: missing inputs, adjusted latency, or no licence.
| Failure | What it looks like | What to check |
|---|---|---|
| Contamination | Near-duplicate test items in training | Overlap check with exact and sentence-level matching |
| Selection | Many private variants, one public score | Pre-registration, held-out tier, disclosure of attempts |
| Construction | Model-generated gold, synthetic cases | Who wrote the cases, who verified the gold |
| Reporting | No inputs, adjusted latency, no licence | Artifact completeness and measurement method |
Contamination and overlap checks
The AutoTrust run repository is unusually explicit. It publishes a training-overlap check that reports no whole suite state, instruction or option in the training data, with zero sentence-level hits in yuri_v3 and openjev_v2. It also reports 547 yuri_v1 rows that share Wikipedia sentences but carry uniform placeholder labels. That distinguishes a real overlap from a shared-source sentence.
The same repository shows the limits. Only engine outputs are published, as compact result rows without benchmark inputs, so the suite must be rebuilt with the index kit. The batched phase reports total_wall_ms as batch wall time divided by 64 rather than per-request latency. The JEV-27B run repeats the same recipe on Qwen/Qwen3.8-27B. That changes what the scores mean.
JevBench handles contamination with a held-out tier: 220 hard decisions, 111 public and 109 held out, written by two named frontier models, cross-reviewed, frozen and hashed before any system ran. Only aggregate sealed statistics are published for the held-out half. The public half can still be selected against, and API rows are operator endpoints that received item text without answer keys.
JevBench Metamorphic is public, synthetic and English, with model-checked gold. Its card says the test set is public, the cases synthetic, the gold verifier a model; a model trained on it would not be measured fairly. It also warns that coherence is not correctness, because a model that answers uniformly scores higher than every evaluated model.
Selection is enough
TypeSafe's anti-benchmaxxing essay makes the argument that a public eval can select the model even when nobody trains on the test. Trying many settings and keeping the best public score is still selection. The essay's examples are Meta's Llama 4 LMArena variants, where 27 private variants were tested and the best was picked, and a Claude vending-machine run.
The essay also describes how public evals shape perception. When GPT-6 Astra launched, Artificial Analysis's Intelligence Index had it tied with GPT-5.6 Sol; the next day a revision put Astra four points ahead, and three days later another tied it with Claude Fable 5.1. The author calls the changes defensible but says the sequence shows intuitions helping decide whether an eval looks valid. The GLM-5.2 web-design leaderboard is the other example: Design Arena's own analysis found more repeatable templates, 25% more code, twice the time, and losses in several design categories.
The caveat is that this is an opinion essay dated 11 September 2026, not a new measurement of Jev. Its value is the mechanism, not the anecdotes. TypeSafe's stated policy follows from it: no standard benchmark table on model releases, dated snapshots that are retired once posted, and internal evals published with caveats and evidence that looks bad for the model. That policy is itself a claim to audit, not a substitute for one.
Vendor numbers and independent numbers
The jev-ai.org audit is a useful model of how to read a vendor claim. It notes that TypeSafe deliberately publishes no MMLU or LMArena scores and plans only one-off evals when the product changes. It then puts the four-workflow average beside the alternatives: Jev at 67.8% for $0.0004 and 0.4 s per case, against 74.1% for "sol" at $0.0836 and 23.3 s. Against similar-accuracy opponents, the article estimates Jev is about 76 times cheaper and 25 times faster than "terra," or 8 times cheaper and 32 times faster than "luna." It flags the weakest workflow, invoice processing at 61.8% versus 79.1%, and notes the homepage demo is one short query TypeSafe says flatters Jev.
The authors run jev-ai.org, a playground and API built on Jev, and state that interest. The System One Models roundup is a broader compilation. It says none of the tests is peer reviewed and most are single runs by one author. On the community Decision Index, Jev is third of 117 rows at 60.11, 2.64 points behind pplx-decider v1.1 at 62.75. On the separate Artificial Analysis index, Jev was first of 11 at launch and second of 13 after Liquid's d1-ultra was added.
The independent 37-dataset evaluation is the strongest counterweight to a vendor-only view. It runs jev-1.13.0 zero-shot on 37 datasets for 346,009 requests under USD 10, reports 95–99% on IMDB, SST-2, HellaSwag and ARC, 86.7% on Belebele across 122 languages, and beats Qwen3.8-27B on 27 of 37 datasets. Its own caveat is that thresholds tuned on training data may not transfer and that all three models degrade on low-resource languages, fine-grained or noisy labels, and rubric judgments. It is an outside benchmark of a vendor model.
The evidence audit
The early evidence audit reviewed 28 papers posted between 19 and 24 September 2026, nine days after Jev's release. It concluded that the typed readout had not shown an independent accuracy advantage over comparable label-probability readouts, and that Jev's clearest gains were latency and cost. It derives a 14-item checklist from recurring weaknesses in the first wave of studies.
It is an early map, not a settled assessment. The corpus covers nine days and one hosted model, but the checklist turns skepticism into specific questions about baselines, labels, thresholds and reporting.
TypedBench illustrates the reporting problem in a single paper: the hosted-model findings describe one unnamed service, and the abstract does not report its cost per item. The benchmark can still be careful, as its noise floor suggests, while leaving a reader unable to map the result to a purchasable system.
A checklist for reading a benchmark
- Who ran it, and who paid for it?
- Was the evaluation pre-registered, and was the hard tier held out?
- Is there a contamination or overlap check, and what exactly did it match?
- Are the baselines paired and current, or is the comparison a vendor reference run?
- Are intervals reported for the headline difference?
- Are cost and latency measured, or adjusted by assumption?
- Are inputs, code and a licence available?
- Are negative results and failed cases reported?
- Has anyone independent rerun it?
Together they separate a result you can act on from one you can only quote. The companion is measuring Jev for field reports and what the papers found for the preprint literature. The launch chart controversy is a case study in how a reporting choice changes the story. The benchmark claims, leaderboard and oracle floor entries define the terms.
