Lies, Damned Lies, and Benchmarks

That trying many settings and keeping the best public score still selects the model even when nobody trains on the test, citing Meta's Llama 4 LMArena variants and a Claude vending-machine run as benchmark-shaped behavior.

No. It is an opinion essay dated 11 September 2026, and the caveat says the incidents are the author's account rather than a new measurement of Jev.

Because the field wants to measure general intelligence but can only optimize what it can see, so public evals are the streetlight. Post-training then pushes capability fastest in the illuminated areas while reliability outside them can stay flat or even get worse, and the essay's sharper claim is that many of the spikes in jagged intelligence are the benchmarks themselves.

That Meta's Llama 4 allegedly ranked near the top of LMArena while not being the public model: Meta was caught testing 27 private variants and picking the best, which had the unusually verbose and emoji-heavy style Arena users rewarded, and the ordinary version later landed far lower. The essay's footnote names the mechanism directly: this is p-hacking, trying enough training choices and then reporting the result that looks strongest.

That benchmarks chase users. When GPT-6 Astra launched, Artificial Analysis's Intelligence Index had it tied with GPT-5.6 Sol, which clashed with the widespread belief that Astra was a major leap; the next day a revision put Astra four points ahead, and three days later another tied it with Claude Fable 5.1 for first. The author does not think the index was rigged and calls the changes defensible, but says the sequence shows people's intuitions about which model is better helping decide whether an eval looks valid.

Since existing benchmarks do not apply to a new type of model, TypeSafe says it is choosing the clean slate rather than a wall of evals showing it beats everyone. Its model releases will carry no standard benchmark table. New evals will be dated snapshots that are immediately retired once posted instead of being hill-climbed, and its evolving internal evals will be published as current best guesses alongside the caveats, any cherry-picking, and evidence that looks bad for them.

When the Chinese open-weight model GLM-5.2 beat Fable 5 on one web-design leaderboard, it became evidence that Chinese models had caught the frontier. Design Arena's own analysis was narrower: GLM used more repeatable templates, generated 25% more code, took twice as long, and still lost to Fable on several design categories. One leaderboard read one way is the essay's version of the streetlight effect.