Article
CommunityJev Media & Talks
jev-ai.org: what TypeSafe's Jev benchmark numbers actually show
A playground operator's audit of every published Jev number: TypeSafe's four-workflow evals, the favourable homepage demo, independent tests and its own calls. It concludes the price and speed are real and the intelligence claim is roughly mid-tier frontier on decision workflows.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Author
- Jev AI
- Last verified
- Oct 2026
Highlights
- The audit says TypeSafe deliberately publishes no MMLU or LMArena scores and plans only one-off evals when the product changes.
- TypeSafe's four-workflow average: Jev at 67.8% for $0.0004 and 0.4 s per case, against 74.1% for 'sol' at $0.0836 and 23.3 s.
- Against similar-accuracy opponents the article estimates Jev is about 76x cheaper and 25x faster than 'terra', or 8x cheaper and 32x faster than 'luna'.
- The homepage demo, $0.000081 in 0.114 s versus $0.013880 in 8.566 s, is one short dense query that TypeSafe says flatters Jev.
- The site's own six production-shaped calls cost $0.000023 to $0.000038 each, with median round trip about 310 ms.
Watch out
Copy reviewed against the published article (30 September 2026) on 2026-10-17. The authors run jev-ai.org, a playground and API built on the Jev model, so they state a practical interest; numbers not produced by TypeSafe come from a single independent developer, a Vercel engineer quoted via TechCrunch and the site's own six calls.
Questions about this entry
Short answers drawn from the linked project or page.
The article says TypeSafe deliberately chose not to publish performance against public benchmarks and plans only one-off evals on product updates; it argues Jev cannot write text, so free-form benchmarks do not apply in their usual form.
Four code-written workflows, security incident triage, agent trace review, invoice processing and customer service, ask narrow questions; accuracy is how often a model's final actions match an average of GPT-6 Astra and Claude Fable 5.1 answers.
A four-task average of 67.8% agreement at about $0.0004 and 0.4 s per case, against 74.1% for 'sol', 73.1% for Opus 5, 67.9% for 'terra' and 53.6% for Claude Haiku 4.5.
Invoice processing, where Jev scored 61.8% against 79.1% for 'sol'; the article says TypeSafe does not explain the gap and treats it as a reminder to test document-heavy workflows on your own data.
The article notes TypeSafe does not name the comparison model, and against similar-accuracy models the gap is smaller: about 76x cheaper and 25x faster than 'terra', or 8x cheaper and 32x faster than 'luna', though still large.
A developer's zero-shot classification pilot scoring 0.910 and 0.870 on two datasets and 0.480 on a third with 300 examples, a Vercel engineer's 5-18x speedup report via TechCrunch, and the site's own six calls at about 310 ms median round trip.
Its own team wrote the workflows, OpenAI and Anthropic models supplied the reference answers, LLMs ran through TypeSafe's structured wrapper, timing came from US West Coast laptops, and TypeSafe calls the headline multiples the higher end of real-world gains.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: Simon Willison's overview of Noul, Choice, and Score, and where a decision model fits.
benwerd53 pointsJev introduces a new shape of LLMRead the thread on Hacker NewsHacker News: swyx posts the Latent Space interview with Diogo Almeida.
swyx4 pointsJev: System One Models for Prod, Not God – With Diogo Almeida, CEO, TypeSafe AIRead the thread on Hacker NewsX: Diogo Almeida in five minutes
X: LLMs vs. Jev, clearly explained
More Jev resources like this
AICrier's news-feed summary of Celeris-1 Decision: a multimodal diffusion model for bounded structured decisions that Celeris says reaches 80% on JevBench, 7 points above Jev 1.13.0, with 67 ms median latency.
Jev Media & TalksArticle#community#press#competition
Community
Fortune reports Jev went viral among developers, with nearly 13% of Vercel's paid teams trying it within a day of its gateway arrival. Three weeks after Jev's launch, OpenAI rolled out a competing Decisions API built on GPT-6 Luna.
Jev Media & TalksArticle#community#press#competition
Community
Adrian Bridgwater's September 21 report on TypeSafe's launch pitch: a model that does not write but decides, a new architecture and RLCD training, and Vercel's Jev adoption numbers through its AI Gateway.
Jev Media & TalksArticle#community#press#explainer
Community
Langfuse's September 18 post presents Jev as a decision model for routing, classification and eval verdicts, and walks through scoring production traces with Jev-as-a-judge in Langfuse.
Jev Media & TalksArticle#community#press#explainer
Community
LangChain's September 25 post argues Jev's typed decisions fit LangGraph's deterministic orchestration, with a litigation review example and a Stagehand browser case where act() latency fell to 0.46 seconds.
Jev Media & TalksArticle#community#press#explainer
Community
A builder's walkthrough of System One, a dual-process browser agent that resolves routine steps with four parallel Jev questions in 70-150ms and falls back to Gemini or Claude below 0.55 confidence.
Jev Media & TalksArticle#community#press#explainer
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.