So are Jev's calibrated probabilities just broken?

Not according to the author, and the wording is careful: these results are specific to the prompts and conditions in this repository and do not establish that Jev probabilities are generally unusable. The documented finding is narrower, that reported confidence did not reflect known uncertainty on tasks where no information existed, while accuracy stayed near chance exactly as it should.

Is the low accuracy on the die the finding, or a bug?

It is the expected part. A fair die cannot be predicted, so 19.0% accuracy across 400 trials is roughly what guessing produces, and the author says the issue is not that Jev failed to predict a random event. The notable result is that those same calls reported a mean probability of 82.9% for the option they selected.

Does it matter whether I ask with Choice or with Noul?

Yes, by a lot. The die asked with Choice reported 82.9% against a 16.7% reference, and a fair coin 92.0% against 50.0%, while the same die asked with Noul reported 19.2%. Noul still over-reported in the low range at 15.0% for twenty equiprobable options, but that is a different order of behaviour from Choice.

Can I rerun these experiments, and what will it cost me?

The offline path is free: Python 3.10+ with no extra packages runs the analysis script over recorded outputs and writes CSV, JSON and Markdown summaries, and the Node preview builds a request from an example file without a key or an install. Only the execute flag makes billable calls, which need a Vercel AI Gateway key since Noul is expressed as a boolean type in that SDK, with 20 second attempts and up to 3 per case.

How do I rerun or just reanalyze these experiments offline?

Python 3.10+ can run python3 scripts/analyze.py over the recorded data with no API calls. API runs need Node 22.16+, npm ci and a Vercel AI Gateway key, and only --execute enables billable calls.

What did the forecast-document experiment show?

Choice did not preserve the uncertainty stated upstream: documents just below and above 50% produced sharply different reported probabilities, 6.6% for a stated 45% and 95.9% for a stated 55%.

CommunityJev Practices & Patterns

Jev Does Not Play Dice

Q: Are these results specific to the author's prompts?

Yes. The README says the results are specific to the prompts and conditions in the repository and do not establish that Jev probabilities are generally unusable.

Recorded experiments on how Jev reports probability rather than whether it predicts well. On a hidden fair die it picked an option with a mean reported probability of 82.9% while accuracy stayed at 19.0% over 400 trials, and Choice lost a stated 45% versus 55% distinction in an input document.

See highlights

Open github.com

Stars: 3
Language: JavaScript
License: MIT
Published by: Community
Author: KantaHayashiAI
Last verified: Oct 2026

KantaHayashiAI/jev-does-not-play-dice

User repository on GitHub

View on GitHub

Experiments on Jev’s probability calibration, uncertainty reporting, and forecast probability preservation.

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Code, recorded outputs, and analysis scripts for probability-output experiments with Jev: fair random draws, Noul (Yes/No) questions, and forecast documents. Write-up: Jev Does Not Play Dice: 83% probability, 19% accuracy on a hidden fair die roll (English) / Jevはサイコロを振らない (日本語, original).

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: JavaScript

Highlights

Fair six-sided die with Choice: mean reported probability 82.9% against a true 16.7%, with observed accuracy of 19.0% over 400 trials.
Noul tracked the reference far better, reporting 19.2% for the die and 15.0% for 20 equiprobable options against a 5.0% reference.
Choice also lost stated uncertainty: a document saying 45% produced 6.6% reported and one saying 55% produced 95.9%.
Offline analysis needs Python 3.10+ with no extra packages, and request previews need neither an API key nor npm install.

Watch out

The recorded responses do not carry an exact model revision, and the README places them in the launch window when jev-1.13.0 was the only publicly available Jev version, so behaviour may have moved since. The author is explicit that these results are specific to the prompts and conditions in the repository and do not establish that Jev probabilities are generally unusable. Choice is the primitive that misbehaves: Noul came close to the reference probability in some conditions while still over-reporting in the low range. Forecast-document percentages are stated values, not independently verified event probabilities. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-column-race

Live race rig that labels 1,000 real app reviews with Jev typed questions against Gemini JSON output on identical batches, then replays recordings and re-ranks rows locally without extra model calls.