Jev Beats Gemini Flash Lite on a Classifier Eval

The post does not say. Saturating an eval usually means the score reached or passed its ceiling, which is a statement about the rubric rather than an accuracy figure you can compare against anything. No fixture count, pass threshold, or per-class breakdown appears in the thread.

Unspecified. One figure is given, 6x, with no baseline latency, no concurrency level, and no statement of whether the comparison includes request construction or only the classifier call. A reply asks the obvious follow-up, run it against 3.8 Flash, which has not been published.

Vercel's own, an internal classifier eval that had previously run on Gemini 2.5 Flash Lite. Nothing about the dataset, labels, or harness is published, so the result cannot be reproduced, and the strongest defensible claim is that one team's existing eval became easier.

Unknown from this post. A saturated eval says the task was too easy for both models, which is at least as likely to mean the fixtures were not discriminative as it means the model is better. The honest takeaway is the speed number plus a reason to run your own eval.

Malte Ubl, the CTO of Vercel, whose bio reads self-driving infrastructure. The post is a short first-person report from an outside engineering team, which is why it carries weight despite the missing methodology.

Malte followed up with And now on AI Gateway, and Vercel Developers announced Jev on AI Gateway with an evaluate snippet using model typesafe-ai/jev. So the eval post was also a run-up to the gateway listing.

Posted 16 September 2026 at 5:02 PM, the post shows 222K views with 40 replies, 36 reposts, 1.1K likes and 222 bookmarks, and one reply simply asks to run it against 3.8 Flash.