Article
CommunityJev Guides & Articles
JevResearch: Jev is not frontier, but still worth your attention
An independent hands-on evaluation that ran Jev live on 16,379 benchmark requests and reports it is no frontier model but genuinely useful for cheap real-time sub-frontier judgement, with 82.7% MMLU-Pro and 76.5% GPQA Diamond.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Author
- JevResearch
- Last verified
- Oct 2026
Highlights
- Jev scored 82.7% on 12,032 MMLU-Pro questions, 76.5% on GPQA Diamond and 21.9% on Humanity's Last Exam multiple choice.
- It answered 16,379 planned requests in three frozen suites with only 19 contract-invalid responses, at about 73 ms of proxy-reported upstream service time.
- The report's category for Jev: cheap real-time sub-frontier judgement such as routing, rubric grading and control-plane decisions.
- It argues 'cannot hallucinate' means Jev cannot return free text; a contract-valid answer can still be wrong, which makes errors hard to notice.
- The report attributes Jev's speed to prefill-dominated scoring without autoregressive decoding, inferred from the closed API's observable behavior.
Watch out
Copy reviewed against the published report (revision 5, 2026-10-03) on 2026-10-17. The evaluation is independent but single-author; Jev was run one-shot with no chain of thought, tools or retries, the architecture discussion is inference from a closed API whose weights were never seen, and benchmark item text is third-party licensed and not republished.
Questions about this entry
Short answers drawn from the linked project or page.
An independent hands-on evaluation that ran Jev live on 16,379 planned benchmark requests across three frozen suites: MMLU-Pro, ARC-Challenge, GPQA Diamond, Humanity's Last Exam multiple choice, MATH-500 subsets and ARC-AGI-2, pinned to service-reported jev-1.13.0.
82.7% MMLU-Pro, 97.9% ARC-Challenge, 76.5% GPQA Diamond, 21.9% HLE multiple choice, 83.1% MATH-500 as multiple choice, 13.6% MATH-500 digit read-out, 59.6% ARC-AGI-2 per cell and 0.0% ARC-AGI-2 exact grid.
It misses about a quarter of graduate science questions and scores a mere 21.9% on Humanity's Last Exam, and the report says where Jev looks frontier-like, such as 97.9% on ARC-Challenge, that is only on a race that finished years ago.
It says Jev cannot return free text because it selects among caller-supplied options, so there is no prose to hallucinate; but a contract-valid answer is not a true answer, and Jev is frequently wrong in beautifully formatted output.
About 73 ms of proxy-reported upstream service time per question, a full graduate-scale benchmark run for cents, and only 19 contract-invalid responses across roughly 16,000 benchmark requests.
Its leading theory, inferred from a closed API, is a small transformer language model post-trained for judgement with its generation head replaced by a probability read-out over caller-supplied options, scoring everything in one batched prefill pass.
It calls Jev a smaller, humbler model that is nonetheless genuinely useful for a job nobody else serves quite this way: cheap real-time sub-frontier judgement like routing, rubric grading and control-plane decisions.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: Simon Willison's overview of Noul, Choice, and Score, and where a decision model fits.
benwerd53 pointsJev introduces a new shape of LLMRead the thread on Hacker NewsHacker News: swyx posts the Latent Space interview with Diogo Almeida.
swyx4 pointsJev: System One Models for Prod, Not God – With Diogo Almeida, CEO, TypeSafe AIRead the thread on Hacker NewsX: Diogo Almeida in five minutes
X: LLMs vs. Jev, clearly explained
More Jev resources like this
Rogue Security's post uses Jev and System One models to frame agentic security as a decision problem: the same send_file action should be allowed, blocked or reviewed depending on who requested it and where the data goes, illustrated with policy case files rather than product results.
Jev Guides & ArticlesArticle#community#press#security
Community
Ian Maurer's ThinkThen analysis argues Jev's faster and cheaper are measurable today while its better is a bet on RLCD: unlike RLHF, which pleases humans, and RLVR, which optimizes benchmarks, Jev is trained to be reliable for programmatic use, in Diogo Almeida's own framing.
Jev Guides & ArticlesArticle#community#press#explainer
Community
A podcast episode on Jev as a decision model rather than a chat model: hosts Luis Calderon and Carina discuss roughly 20x faster Coach testing, why that preliminary figure differs from TypeSafe's vendor benchmarks, and why type safety does not guarantee a correct decision.
Jev Guides & ArticlesArticle#community#press#podcast
Community
A BIG CHANGE analysis of the Diogo Almeida Latent Space interview: Jev's most consequential proposition is the unit of automation, a bounded judgment inside an existing process, and the launch claim that Jev can't hallucinate needs a narrow reading because schema matching does not establish truth.
Jev Guides & ArticlesArticle#community#press#explainer
Community
TWIML episode 779 features TypeSafe co-founder and CEO Diogo Almeida on Jev, the company's model for bringing fast, reliable intelligence directly into software, and why text-generating models are poorly suited to many real-world automation decisions.
Jev Guides & ArticlesArticle#community#press#podcast
Community
On The MAD Podcast, Matt Turck talks with TypeSafe CEO Diogo Almeida about why intelligence per dollar matters more than enterprise adoption, and why RLVR benchmark gains still leave models 95% Einstein, 5% Mr. Bean.
Jev Guides & ArticlesArticle#community#press#podcast
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.