Ran @typesafeai's Jev against an existing classifier eval that previously used Gemini 2.5 Flash Lite. It won both on quality (saturated the eval) and speed (6x)
Jev Beats Gemini Flash Lite on a Classifier Eval
Vercel CTO Malte Ubl reports that Jev beat an existing classifier eval previously run on Gemini 2.5 Flash Lite, saturating the eval on quality and running 6x faster.
- Category
- Practices & Patterns
- Published by
- Community
- Author
- Malte Ubl
- Added
- 2026-09-20
The post
Tagscommunityxevaluationbenchmarkvercel
Highlights
- Says Jev saturated the eval that Gemini 2.5 Flash Lite had not.
- Claims a 6x speedup on the same task.
- A head-to-head eval report from an outside engineering team.
Watch out
One internal eval with no published methodology or sample size.
More like this
Yuchen Jin's take on Jev for LLM-as-a-judge work: 20-200x faster and 40-400x cheaper, with a screenshot of the model picking which candidate is AGI after a prompt sweep under $0.10.
Practices & Patterns#community#x#evaluation
Community
The WebMCP benchmark author reports that Jev paired with the small Mercury 2.5 model solved 100% of tasks at roughly 112x lower model cost than GPT-6 Astra with computer use.
Practices & Patterns#community#x#benchmark
Community
A map of documented Jev use cases, routing, guardrail verification, citation checking, reranking, rubric scoring, bulk classification, and semantic linting in CI, each with a confidence policy.
Practices & Patterns#community#routing#evaluation
Community
From the community
Posts from builders shipping with Jev right now.
A Claude session from 1M to 86K tokens
This is actually insane. This uses @typesafeai Jev model, as a plugin in Claude to review all the un-nesseasary tool calls, and it takes 1s to run! Like, literally, 1 second to take my Claude session from nearly 1M to ... 86K tokens! 😮 Ask your claude to install it and be Show more
found the perfect use case for @typesafeai Jev: instant compaction in 2026, why is compaction still a summarization prompt? Jev can make it instant by scoring every tool call and dropping what’s irrelevant
Vercel's fx safety reviewer, 18x faster
We're seeing extraordinary results from @typesafeai. Default mode in 𝚏𝚡 is auto, with a safety reviewer analyzing every command. That reviewer runs on GPT Luna today. Jev is up to 18x faster (p95) *and* more accurate. It's coming to @vercel AI Gateway and likely new default.
@fazxes
We benchmarked fx auto mode (safety) classifier with @typesafeai's Jev. tl;dr: ~5-18x faster and more accurate than 𝚐𝚙𝚝-𝟻.𝟼-𝚕𝚞𝚗𝚊, our current top choice
