TypeSafe's Jev Can't See. I Made It Guess What I Drew Anyway — news.ycombinator.com

Four hundred drawings, preregistered before running, according to the linked write-up on mikulskibartosz.name. The captured thread adds nothing beyond the submission, so the protocol details exist only in the article.

The headline failure is the airplane: Jev collapses on out-of-distribution sketches of that class while Sonnet holds. The per-class accuracy split is not in the thread, and the article's game UI is separate from the benchmark tables.

Arguably not. Jev only ever sees text describing an SVG, never the drawing, so a collapse on an unseen shape class may reflect the state encoding rather than classification skill. The thread does not discuss that confound.

Not on their own. One author, one benchmark, modest thread size, and no documentation of pricing, latency, or how many questions ride in a call. Reproduce it on your own label distribution first.

Bartosz Mikulski, published on mikulskibartosz.name. The captured Hacker News thread is just the submission, drawing 3 points with no discussion added.

Jev cannot take images, so the test feeds it text describing an SVG and has it guess what was drawn. That is why a collapse on an unseen shape class is hard to separate from the state encoding rather than classification skill.

They are separate. The game UI is a demo, while the benchmark tables hold the measured 400-drawing comparison; the entry warns not to read the UI as part of the measured result.