CommunityJev Guides & Articles
Jev hands-on against Sonnet 5
Five self-funded experiments on jev-1.13.0 from a Mac mini, comparing direct HTTP with claude-sonnet-5 after an earlier test mistook CLI startup for model speed.
- Format
- Article
- Published by
- Community
- Author
- zhuermu
- Last verified
- Sep 2026
Used inJev for Software & DevTools
Highlights
- 20 September 2026. About 310,000 input tokens and $0.013, on a Mac mini M4 with stdlib Python.
- Four judgments in one call: Jev p50 1,412 ms versus claude-sonnet-5 p50 3,237 ms, three runs.
- After subtracting each arm's idle floor, the judgment increment is 268 ms versus 923 ms.
- Same-day revision recomputed cost from measured tokens, from 12–216× down to 4–79×.
- Skill routing is labeled a fully negative result. The first latency test had counted CLI cold start.
Watch out
Comparison-arm model usage sits on separate subscriptions and is not in the $0.013. Not sponsored.
Questions about this entry
Short answers drawn from the linked project or page.
Against the API, with both arms on stdlib urllib hitting HTTP directly, p50 over three runs. The author flags that his first attempt compared CLIs and mistook cold start for model speed. Four questions in one call give 1,412 ms against 3,237 ms, and subtracting each arm's idle floor narrows it to 268 ms against 923 ms. Read the second pair.
None of it. The comparison arms' model usage runs on separate subscriptions and is excluded from the $0.013. Only Jev's roughly 310,000 input tokens are counted, at the published $0.042 per million with output tokens free, so this is not a like-for-like total spend figure.
No. It is labelled a fully negative result in the post, and the write-up says so rather than burying it. That is the useful part: the cheap wins were scoring and routing-style judgments, not deciding which of an agent's own skills to load.
No. Several questions over the same state go out in one request, are evaluated in parallel, and cannot see each other's answers, so a judgment has to be decomposed into independent dimensions. Only a real dependency justifies a second request. Input is text only, and state plus the longest question must stay under 32k.
The write-up treats it as the real differentiator. An LLM returns a yes or no without saying how sure it is; you can ask it to fill in a confidence field, but that is a number it reports about itself. Jev's confidence comes from the concentration of the probability distribution, and the post says that routable uncertainty is the one property nothing else in the comparison offers.
The published specs are $0.042 per million input tokens with output tokens free; 250,000 tokens per second and 1,200 requests per minute, documented as adjusting dynamically; and a 64k context per request where state plus the single longest question must stay under 32k. Input is text only, as a string, JSON object or array, and several questions over one state go out in one request.
Self-funded with no relationship to TypeSafe, run on 2026-09-20 on a Mac mini M4 with zero-dependency stdlib Python hitting the HTTP API directly. jev-1.13.0 was reached through the jev-latest alias; comparison arms were claude-sonnet-5 over direct HTTP plus three CLIs. What the author paid was $0.013 for roughly 310,000 input tokens, and the comparison arms' model usage sits on separate subscriptions excluded from that total.
More Jev resources like this
Datadog's walkthrough of one Jev rubric that scores every criterion in a single request, then runs as online evals on live spans and as offline experiments.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
Alex Molas argues that a probability fit on TypeSafe's data is not calibrated on yours, so Jev's numbers should be treated as scores.
Jev Guides & ArticlesArticle#community#article#calibration
Community
Latent Space's AINews roundup of launch day: the Jev thread atop Hacker News, the RLCD decision-model framing, and community readings that compare it to DSPy-style typed signatures for routing and scoring.
Jev Guides & ArticlesArticle#community#article#models
Community
Opper hosts Kev 4B and typesafe/jev-latest on the same /v1/systemone compat route and compares accuracy, tokens, and latency on 362 post-release questions.
Jev Guides & ArticlesArticle#community#article#open-models
Community
Thread on Kartik Pansuriya's essay arguing calibrated probabilities matter more than headline accuracy for production classifiers.
Jev Guides & ArticlesPost#community#calibration#evaluation
Community
Sam Reghenzi's product-taxonomy benchmark: a Jev Choice at every level versus a gpt-5.2 agent that descends the tree and can be sent back by a judge.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
The typed-answer shape removes a category of data risk, not all of it. Zero data retention per surface, the legal set to read, and the controls that remain yours to run.
Jev Guides & Articles#privacy#compliance#legal
Read article
TypeSafe serves Jev from one API. A second tier of independent sites wraps it with their own keys, credits, and playgrounds. How to read the disclaimers before you paste a key.
Jev Guides & Articles#ecosystem#trust#hosting
Read article
The first ten days produced three receptions that rarely meet: a Product Hunt launch, a 620-point Hacker News thread, and a r/LocalLLaMA that doubted the framing. Reading them together.
Jev Guides & Articles#launch#community#reception
Read article
From the Jev community
Posts from builders shipping with Jev right now.