Article
CommunityJev Guides & Articles
Blackwell Systems: adversarial diligence on Jev's decision layer
Blackwell Systems' adversarial technical due diligence on Jev, built only from public material and live API calls. The author reproduces TypeSafe's published addressee numbers within a point and finds a real but noise-type-specific robustness moat.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Author
- Blackwell Systems
- Last verified
- Oct 2026
Highlights
- Calling the live model reproduced the published addressee scores within a point: 0.962/0.939/0.933 versus the published 0.962/0.944/0.927.
- Jev holds flat under phonetic, casing and speech-to-text noise, but collapses under keyboard and random-character corruption that drops BoolQ 0.925 to 0.560.
- Off its niche, Jev scored 0.81 on Banking77, below the free open 399M dev-0.4b model at 0.913 measured on the same slice.
- A cheap fine-tuned model reaches about 0.90 on clean judgment against Jev's 0.962, and about 0.79 on misheard input against Jev's 0.927.
- The piece's verdict: the classification result is unchanged, and the noisy-input moat is real but modest, narrower than an earlier draft reported.
Watch out
Copy reviewed against the published post (21 September 2026) on 2026-10-17. It is independent analysis from public materials by a consultancy, not a paid engagement; several figures come from the author's own runs, the cross-domain noise results used a simpler request format and are called directional, and the post revises its own earlier draft's numbers.
Questions about this entry
Short answers drawn from the linked project or page.
Jev as a decision layer, via a worked example of adversarial technical due diligence: it checks TypeSafe's published numbers against the live model, probes noise robustness, and compares Jev against cheap owned alternatives on general decision tasks.
Yes. Calling the live model on the same 237-decision addressee set gave 0.962/0.939/0.933 (clean/stt/misheard) against the published 0.962/0.944/0.927, within a point on each, with recall in the published band.
It is noise-type-specific: Jev holds nearly flat under phonetic, casing and speech-to-text corruption across three domains, but collapses under keyboard and random-character noise, which the author identifies as the axis a competitor could still win.
On Banking77 it scored 0.81 against 0.913 for a free 399M open model measured on the same slice; CLINC150 and SNLI are compared against published fine-tune ranges of roughly 0.95-0.97 and 0.90-plus, which the post calls absolute standing rather than a same-harness contest.
It narrows the gap a lot on clean input, reaching about 0.90 against Jev's 0.962, but on noisy misheard input a cheap owned model reaches about 0.79 against Jev's 0.927, leaving Jev roughly 0.13 ahead.
No. The post reports that the explicit phonetic-match feature did not help because the model already learns the phonetic bridge from the data, so the noisy-input edge survives as a real but modest moat.
It says Jev's value concentrates in its specialized niche; on general decision tasks an owned or open model can be both cheaper and more accurate, and running the experiment reversed the author's earlier judgment-side build recommendation.
More Jev resources like this
Interactive comparison of 21 decision models, with direct-provider and OpenRouter routes measured separately on 1,100 public jev-bench items. The charts plot accuracy against benchmark cost, official prices and median latency, with Jev 1.13 and 1.14 measured separately.
Jev Guides & ArticlesArticle#community#press#calibration
Community
The desk's index of every Jev measurement it has published, keeping each figure's original caveat: JevBench boards, Vercel gateway reach, Celeris and Artificial Analysis results, and the desk's own Jev Day probes which it says it did not rerun.
Jev Guides & ArticlesArticle#community#press#calibration
Community
Side-by-side tables placing similar Jev tests next to each other, from the JevBench API board to Vercel paid-team reach, with the desk's warning that differing sets, labels and comparators mean the rows are not a ranking.
Jev Guides & ArticlesArticle#community#press#competition
Community
A podcast episode on Jev as a decision model rather than a chat model: hosts Luis Calderon and Carina discuss roughly 20x faster Coach testing, why that preliminary figure differs from TypeSafe's vendor benchmarks, and why type safety does not guarantee a correct decision.
Jev Guides & ArticlesArticle#community#press#podcast
Community
Braintrust's provider page covers two ways to use TypeSafe Jev: instrumenting System One calls as decision spans, and running Jev as an LLM judge that turns typed choices into scores or labels.
Jev Guides & ArticlesArticle#community#press#explainer
Community
Langfuse's guide to decision-model evaluators runs OpenAI's gpt-6-luna or TypeSafe Jev over observations, where each Choice, Score or Yes/no question becomes one Langfuse score.
Jev Guides & ArticlesArticle#community#press#explainer
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.