Post
CommunityJev Practices & Patterns
Jev Does Not Play Dice — kantahayashiai.github.io
Kanta Hayashi's calibration check using a hidden fair die: across 400 trials Jev picked '1' every time at an average 83% probability while being right 19% of the time, and handing forecast documents through Choice shrank a stated 30% risk to about 5%.
- Format
- Post
- Reading time
- 3 min
- Published by
- Community
- Author
- Kanta Hayashi
- Last verified
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is Jev Does Not Play Dice — KantaHayashiAI.
Highlights
- 400 hidden fair-die trials via Choice: '1' chosen every time at 82.9% average probability, hit rate 19.0% (76/400).
- The reported probability never fell below 71% (median 83%) across the trials, more than four times the true 1/6.
- Noul was closer on the die (19.2% vs 16.7%) but drifted high as options grew: 15.0% mean against a true 5.0% with 20 destinations.
- Forecast documents read through Choice rewrote a stated 30% shortage risk to 5%, 5% and 6% (mean 5.3%); Noul averaged 26.7%.
- The author cites Archer Hume's well-calibrated 1,200-item MMLU sample and argues exam performance and unknowable outcomes are different tests.
Watch out
Page text fetched 2026-10-17. All numbers are the author's own experiments - Choice and Noul calls through the Vercel AI Gateway in the week of Jev's launch, posted September 20, 2026, with code and recorded outputs on GitHub; the original was written in Japanese and translated with LLM help. The confidence figures are noted as consistent with Stanislav Yurin's reconstructed formula, not verified here. The post stresses it is a validation warning, not a claim that Jev is useless.
Questions about this entry
Short answers drawn from the linked project or page.
Over 400 trials with a hidden fair die, Jev's Choice picked '1' every time and assigned that pick an average probability of 82.9%. It was right 19.0% of the time (76 of 400), essentially the theoretical 16.7% chance rate - the choosing was fine, the reported probability was not.
Nobody can guess a fair die, so chance accuracy is expected and always answering '1' is a reasonable fixed tie-break. The problem is that with no information to go on, Jev did not report the uncertainty as a probability: the output should have been near 1/6 per face, not 83% for one face.
Partially. Noul was close on the die (19.2% mean vs 16.7% true) and on the four-way case (24.7% vs 25.0%), but drifted high as the true probability shrank: with 20 destinations and a true 5.0%, every answer sat between 13% and 17%, mean 15.0%.
Synthetic documents stating probabilities were read by Jev before the next step. For three inventory documents stating a 30% shortage risk, Choice returned 5%, 5% and 6% (mean 5.3%) while Noul averaged 26.7%; varying the stated number showed an abrupt jump, with a stated 45% becoming 6.6% and a stated 55% becoming 95.9%.
The author says they do not contradict each other: Archer Hume's 1,200-item MMLU sample showed good calibration on exam-style tasks, and on synthetic tasks close to the official demos the author also saw nothing this extreme. Exam questions test knowing when a question is hard; the die tests knowing when the outcome is unknowable.
Validate probabilities on your own task: group validation cases by the probability the model gave and compare average confidence with actual hit rate. Include edge cases with hidden information, and do not let a model overwrite numbers from a source document if you need to pass them downstream.
Kanta Hayashi wrote it, posted September 20, 2026 as an 8-minute read, with code and recorded outputs in a linked GitHub repository. The post was originally written in Japanese and published on note, then translated into English with LLM help; it credits Archer Hume's MMLU test and Stanislav Yurin's confidence reconstruction.
More Jev resources like this
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Jev Practices & Patterns#community#python#benchmarks
Community
AbdelStark/jev-benchmarksPython
AbdelStark/jev-benchmarksPythonCLI that fits a per-question confidence threshold to a target accuracy on your own labeled data, verifies it on a held-out split, estimates how much traffic still needs an LLM, and re-checks locked thresholds in CI. It publishes no Jev results of its own.
Jev Practices & Patterns#community#python#calibration
Community
abhixhek/jevcalPython
abhixhek/jevcalPythonAn independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.
Jev Practices & Patterns#community#python#routing
Community
FirasSX914/JanusPython
FirasSX914/JanusPythonAn independent measurement spanning 5,477 test rows and 34.1M input tokens, comparing one direct Jev question per row with 12-14 scored dimensions fitted to local labels.
Jev Practices & PatternsArticle#community#benchmarks#evaluation
Community
Sixteen thousand calls against gpt-5.4-mini and gpt-5.6-luna on four public datasets plus production pipeline decisions, with a threshold procedure and an explicit filter-not-replacement verdict.
Jev Practices & PatternsArticle#community#benchmarks#evaluation
Community
A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.
Jev Practices & Patterns#community#typescript#agents
Community
vinilana/jev-eval-agentHTML
vinilana/jev-eval-agentHTMLFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.