CommunityJev Guides & Articles
Jev can't be calibrated — news.ycombinator.com
The Hacker News thread on Alex Molas's calibration argument, including a commenter who reran the fair-coin and fair-die prompts.
- Format
- Post
- Published by
- Community
- Author
- alexmolas
- Last verified
- Sep 2026
Used inJev for Software & DevTools
Highlights
- 65 points on 23 September 2026.
- kantahayashi reports 200 fair-coin calls that returned about 0.92, matching the figure in the post.
- The same commenter reports 400 fair-die calls, with no face shown to the model, that always picked face 1 at about 83%.
- Molas replies that problems whose answer cannot be known in advance are exactly where a calibration claim has to hold.
Watch out
The die and coin counts are one commenter's tests, not a published dataset. Point totals move.
Questions about this entry
Short answers drawn from the linked project or page.
Told a die was unbiased with no result shown, Jev picked face 1 on all 400 calls and attached about 83% to it. A fair coin over 200 calls came back near 0.92. Always picking the same face is defensible; the probability attached to it is the problem.
On tasks whose answer cannot be known in advance, that is what the thread's evidence says. The post's whole argument is that this is exactly where a calibration claim has to hold, since TypeSafe's docs say an outcome given 0.2 should happen about a fifth of the time.
Both readings are in the thread. The counter-test is that when 'not sure' is offered as an option, a commenter got it chosen 100% of the time, and continuous score options proved more accurate than discrete ones, so the failure is sensitive to how you frame the answers.
One commenter's 400 die calls and 200 coin calls, plus a short write-up of their own. It is a pointed reproduction rather than a published evaluation, which is enough reason to test calibration on your own labels before trusting any threshold.
A commenter reports that if it can say not sure it picks that 100% of the time, but if you pin it in a corner it does the strange deterministic thing. The same commenter found continuous options much more accurate than the choices.
A commenter says that if you offer probabilities as the answers, Jev picks the right one with high credence, which is different behaviour from choosing among die faces.
It says a higher probability should correspond to a greater chance that the answer is correct, and that outcomes assigned a probability of 0.2 should occur about 20% of the time, which is the standard the die and coin tests are checked against.
Related terms
Glossary definitions related to this entry.
Other listings of this project
Forks and copies of this project in the directory. This entry is the canonical one.
Alex Molas argues that a probability fit on TypeSafe's data is not calibrated on yours, so Jev's numbers should be treated as scores.
Jev Guides & ArticlesArticle#community#article#calibration
Community
More Jev resources like this
Thread on Kartik Pansuriya's essay arguing calibrated probabilities matter more than headline accuracy for production classifiers.
Jev Guides & ArticlesPost#community#calibration#evaluation
Community
Alex Molas argues that a probability fit on TypeSafe's data is not calibrated on yours, so Jev's numbers should be treated as scores.
Jev Guides & ArticlesArticle#community#article#calibration
Community
Datadog's walkthrough of one Jev rubric that scores every criterion in a single request, then runs as online evals on live spans and as offline experiments.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
Five self-funded experiments on jev-1.13.0 from a Mac mini, comparing direct HTTP with claude-sonnet-5 after an earlier test mistook CLI startup for model speed.
Jev Guides & ArticlesArticle#community#article#evaluation
Community
Latent Space's AINews roundup of launch day: the Jev thread atop Hacker News, the RLCD decision-model framing, and community readings that compare it to DSPy-style typed signatures for routing and scoring.
Jev Guides & ArticlesArticle#community#article#models
Community
958GitHub stars
An independent, community-maintained catalog of open-source projects built with Jev, citing public evidence behind each entry it lists.
Jev Guides & Articles#community#typescript#awesome-list
Community
heyjunpenn/awesome-jevAstro
heyjunpenn/awesome-jevAstroFrom the Jev guides
Original write-ups on what this entry is for.
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
The typed-answer shape removes a category of data risk, not all of it. Zero data retention per surface, the legal set to read, and the controls that remain yours to run.
Jev Guides & Articles#privacy#compliance#legal
Read article
From the Jev community
Posts from builders shipping with Jev right now.