Article
CommunityJev Guides & Articles
Jev News evals desk: every published Jev measurement
The desk's index of every Jev measurement it has published, keeping each figure's original caveat: JevBench boards, Vercel gateway reach, Celeris and Artificial Analysis results, and the desk's own Jev Day probes which it says it did not rerun.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- Jev News
- Last verified
- Oct 2026
Used inJev for Software & DevTools
Highlights
- Puts vendor tables next to independent write-ups and personal clips, keeps each row's original caveat, and states that it did not rerun any of the measurements.
- Notes that TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; the numbers are what people posted.
- The JevBench API board opened October 17, 2026: kettei-fat first of 39 at composite 77.8, with Jev 1.13.0 seventh at 71.5.
- Vercel AI Gateway prints Jev at 35.3% of paid teams over two weeks against Sonnet 5.5 at 18.1%; the September 18 post still shows nearly 13% at 24 hours.
- Celeris-1 Decision scored 80.0% on 3,210 jev-bench items against Jev 1.13.0 at 73.1% on a 22-dataset harness, with a 67 ms median.
Watch out
Copy reviewed against the evals page (updated October 17, 2026) on 2026-10-17. The desk states plainly that it did not rerun any measurement and that TypeSafe's Master Customer Agreement forbids publishing benchmarks of the Services; the figures are what people posted, each with its own caveat.
Questions about this entry
Short answers drawn from the linked project or page.
A running index of measurements the desk has already published, with the caveat that came with each one. Vendor tables sit next to independent write-ups and personal clips, and side-by-side comparisons live on the companion compare page.
No. The page states plainly that it did not rerun any of them; its own probes were limited calls such as a keyless GET of /v1/models that returned HTTP 401, and it says it did not send billed POSTs to most of the APIs.
It notes TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; the numbers on the page are what other people posted, attributed to whoever measured them, not official results.
Opened October 17, 2026 at release v1.6.1: kettei-fat ranks first of 39 API offerings at composite 77.8 and capability 86.3, while Jev 1.13.0 ranks seventh at 71.5 and capability 77.1.
Vercel's October 2026 Production Index launch table puts Jev at 35.3% of paid teams in its first two weeks against Sonnet 5.5 at 18.1%, while the September 18 post still prints nearly 13% at 24 hours and three different spend shares appear.
Celeris-1 Decision at 80.0% on 3,210 jev-bench items against Jev 1.13.0 at 73.1% with a 67 ms median, Aviz Maeir's 180-scene Box test at 82%, the OpenRouter Armor tool-call set, and the desk's own Jev Day 2026 Box and Choice probes.
It warns readers not to merge prints: ImageJevBench 75.4 and the blog's 93.1% are different from JevBench 77.8, RAGTruth rows keep their own tests, and each board's scale, draw and date are called out.
More Jev resources like this
Braintrust's provider page covers two ways to use TypeSafe Jev: instrumenting System One calls as decision spans, and running Jev as an LLM judge that turns typed choices into scores or labels.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Langfuse's guide to decision-model evaluators runs OpenAI's gpt-6-luna or TypeSafe Jev over observations, where each Choice, Score or Yes/no question becomes one Langfuse score.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Blackwell Systems' adversarial technical due diligence on Jev, built only from public material and live API calls. The author reproduces TypeSafe's published addressee numbers within a point and finds a real but noise-type-specific robustness moat.
Jev Guides & ArticlesArticle#community#press#competition
Community
Interactive comparison of 21 decision models, with direct-provider and OpenRouter routes measured separately on 1,100 public jev-bench items. The charts plot accuracy against benchmark cost, official prices and median latency, with Jev 1.13 and 1.14 measured separately.
Jev Guides & ArticlesArticle#community#press#calibration
Community
Side-by-side tables placing similar Jev tests next to each other, from the JevBench API board to Vercel paid-team reach, with the desk's warning that differing sets, labels and comparators mean the rows are not a ranking.
Jev Guides & ArticlesArticle#community#press#competition
Community
A podcast episode on Jev as a decision model rather than a chat model: hosts Luis Calderon and Carina discuss roughly 20x faster Coach testing, why that preliminary figure differs from TypeSafe's vendor benchmarks, and why type safety does not guarantee a correct decision.
Jev Guides & ArticlesArticle#community#press#podcast
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.