Article
CommunityJev Guides & Articles
Vilinski: decision models compared on accuracy, cost and speed
Interactive comparison of 21 decision models, with direct-provider and OpenRouter routes measured separately on 1,100 public jev-bench items. The charts plot accuracy against benchmark cost, official prices and median latency, with Jev 1.13 and 1.14 measured separately.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Last verified
- Oct 2026
Used inJev for Software & DevTools
Highlights
- The comparison measures 21 individual models and routes on the same 1,100 public jev-bench questions (22 datasets x 50 items, seed 7).
- Prices checked 2026-10-15: Celeris-1 Decision at $0.04 input and $0 output, Jev 1.13 and 1.14 at $0.042/$0, GPT-6 Luna Decisions at $0.10/$0.
- Clef routes differ by provider: OpenRouter lists $0.042/$0.021/$0.150 input for Clef, Clef-flash and Clef-omni versus Cloudflare direct at $0.240/$0.038/$0.150.
- Native decision probabilities, DIY token logprobs and Celeris's generative self-reported probabilities are labelled as distinct methods.
- Span-01 and Span-01 Lite are explicitly excluded from the comparison, and intervals use 2,000 seeded bootstrap resamples.
Watch out
Copy reviewed against the published comparison (updated 2026-10-15) on 2026-10-17. It is one author's measurement dashboard, not a peer-reviewed benchmark; timings include process start, TLS, network and retries, failed or unsupported answers score as uniform, and Celeris-1 and Magnus were measured with a JSON probability adapter with thinking disabled.
Questions about this entry
Short answers drawn from the linked project or page.
21 individual decision models, with direct-provider and OpenRouter routes measured separately, on 1,100 items of the public jev-bench (22 datasets x 50 test items, seed 7), one request at a time through the jev CLI.
The horizontal axis can be benchmark cost per 1,000 calls, official input, output or cache prices, API-reported cost, or median latency; the vertical axis can be jev-bench accuracy, yes/no accuracy, Brier score or accuracy on the author's own 60 items.
Because the same model can carry different prices per route: OpenRouter lists Clef at $0.042 input while Cloudflare direct lists $0.240, with dotted lines connecting the same model across routes and solid lines connecting families within a route.
Accuracy is fractional top-option accuracy and Brier is the sum of squared errors over options; failed, refused or unsupported answers score as uniform, and only successful calls are timed.
The page distinguishes native decision probabilities, DIY token logprobs and Celeris's generative self-reported probabilities; Celeris-1 and Magnus are generative models measured through a JSON probability adapter with thinking disabled.
Celeris-1 Decision at $0.04 input with free output, Jev 1.13 and 1.14 at $0.042/$0, GPT-6 Luna Decisions at $0.10/$0, and the deprecated GPT-4.1 mini DIY at $0.40 input, $0.10 cached and $1.60 output.
It is one author's dashboard: timings include process start, TLS, network and retries, partial runs stay out of the plot, both Span-01 variants are excluded, and statistical intervals come from 2,000 seeded bootstrap resamples.
More Jev resources like this
Blackwell Systems' adversarial technical due diligence on Jev, built only from public material and live API calls. The author reproduces TypeSafe's published addressee numbers within a point and finds a real but noise-type-specific robustness moat.
Jev Guides & ArticlesArticle#community#press#competition
Community
The desk's index of every Jev measurement it has published, keeping each figure's original caveat: JevBench boards, Vercel gateway reach, Celeris and Artificial Analysis results, and the desk's own Jev Day probes which it says it did not rerun.
Jev Guides & ArticlesArticle#community#press#calibration
Community
Side-by-side tables placing similar Jev tests next to each other, from the JevBench API board to Vercel paid-team reach, with the desk's warning that differing sets, labels and comparators mean the rows are not a ranking.
Jev Guides & ArticlesArticle#community#press#competition
Community
A podcast episode on Jev as a decision model rather than a chat model: hosts Luis Calderon and Carina discuss roughly 20x faster Coach testing, why that preliminary figure differs from TypeSafe's vendor benchmarks, and why type safety does not guarantee a correct decision.
Jev Guides & ArticlesArticle#community#press#podcast
Community
Braintrust's provider page covers two ways to use TypeSafe Jev: instrumenting System One calls as decision spans, and running Jev as an LLM judge that turns typed choices into scores or labels.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Langfuse's guide to decision-model evaluators runs OpenAI's gpt-6-luna or TypeSafe Jev over observations, where each Choice, Score or Yes/no question becomes one Langfuse score.
Jev Guides & ArticlesArticle#community#press#explainer
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.