CommunityJev Guides & Articles
Jev vs Claude Opus 5 on Banking77
OpenRouter's Banking77 run: 3,080 support utterances, 77 intents, Jev 1.13 against Claude Opus 5 on the same one-line criteria, with latency, cost, and a confidence cascade.
- Format
- Article
- Published by
- Community
- Author
- Kenny Rogers
- Last verified
- Sep 2026
- Platforms
- OpenRouter
Highlights
- Full Banking77 test split, 3,080 utterances, 40 per intent. Criteria were written from the label names.
- Jev 1.13 accuracy 81.0% (79.6–82.3) versus Opus 84.4% (83.1–85.6). They agreed on 89.3%.
- Median latency 175 ms versus 2,266 ms. The full run cost $0.34 versus $7.44. Zero malformed responses.
- Confidence at least 0.99 covers 58% of rows at 96.3% accuracy. Below 0.5, 3.5% of rows, accuracy is 29.6%.
Watch out
OpenRouter's own run on 22 September 2026, through its Decisions API. The post says confidence overestimated accuracy in the mid-range.
Questions about this entry
Short answers drawn from the linked project or page.
A paired bootstrap puts the gap at a 95% interval of 2.3 to 4.4 points, so Opus being ahead is unlikely to be chance. Both models returned zero malformed responses across 3,080 utterances — every error was a wrong label — and both sat below the low 90s a fully fine-tuned encoder reaches.
At its cheapest, yes, which flatters it. The reported Opus cost assumes the 3,700-token system prompt was cached and billed at the cache-read rate; at list price it would have been about $19 per thousand requests. So the 13x speed and 22x cost gaps are the conservative comparison.
As a ranking, yes. As a calibrated probability, no — the post says Jev overestimated its own accuracy in the mid-range. It still separates cleanly: rows at 0.99 confidence or above were 58% of the set at 96.3% accuracy, while the 3.5% below 0.5 were right 29.6% of the time.
Unevenly. Opus leads on 35 of the 77 intents, Jev on 15, with 27 ties. They split on 10.7% of utterances and Opus got 175 of those right alone against Jev's 72, including 80% versus 52.5% on receiving_money, while Jev took compromised_card 95% to Opus's 70%.
Jev's full run cost $0.34, or $0.11 per 1,000 requests, with mean input 2,605 and output 826 tokens, against Opus at $7.44, or $2.42 per 1,000, mean input 3,750 and output 17 tokens. Opus's figure has the 3,700-token system prompt cached, and its list rate would be about $19 per 1,000 without caching.
Jev's median round trip was 175 ms with p95 270 ms and p99 353 ms. Opus, without reasoning mode, was 2,266 ms median, 3,004 ms p95 and 3,835 ms p99, so the slowest Jev call was faster than the fastest Opus call.
Opus leads on 35 of the 77 intents, Jev on 15, and they tie on 27. Opus's biggest lead is receiving_money at 80.0% against Jev's 52.5%, while Jev's is compromised_card at 95.0% against Opus's 70.0%.
More Jev resources like this
Sam Reghenzi's product-taxonomy benchmark: a Jev Choice at every level versus a gpt-5.2 agent that descends the tree and can be sent back by a judge.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
Opper hosts Kev 4B and typesafe/jev-latest on the same /v1/systemone compat route and compares accuracy, tokens, and latency on 362 post-release questions.
Jev Guides & ArticlesArticle#community#article#open-models
Community
Privatemode's write-up of typed decisions from GLM-5.3-Flash in one forward pass, with latency, cost, and accuracy set next to Jev and Laya.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
OpenRouter times Jev 1.13, GPT Luna, and Claude Opus on 60 support tickets: intent plus an escalation flag, then says the reply still belongs to a generative model.
Jev Guides & ArticlesArticle#community#article#routing
Community
The r/LocalLLaMA thread arguing that Jev's constrained probabilities are ordinary zero-shot classification, and that the comparison should be other classifiers.
Jev Guides & ArticlesPost#community#open-models#evaluation
Community
Philipp Acsany's tutorial replaces a Y-or-N input check with Jev, then asks Noul, Score, and Choice in one call through OpenRouter.
Jev Guides & ArticlesArticle#community#article#tutorial
Community
From the Jev guides
Original write-ups on what this entry is for.
The typed-answer shape removes a category of data risk, not all of it. Zero data retention per surface, the legal set to read, and the controls that remain yours to run.
Jev Guides & Articles#privacy#compliance#legal
Read article
TypeSafe serves Jev from one API. A second tier of independent sites wraps it with their own keys, credits, and playgrounds. How to read the disclaimers before you paste a key.
Jev Guides & Articles#ecosystem#trust#hosting
Read article
The first ten days produced three receptions that rarely meet: a Product Hunt launch, a 620-point Hacker News thread, and a r/LocalLLaMA that doubted the framing. Reading them together.
Jev Guides & Articles#launch#community#reception
Read article
From the Jev community
Posts from builders shipping with Jev right now.