A pre-registered adversarial evaluation of jev: nine experiments, 28 predictions fixed before any request was sent, and a report plus prompting guide regenerated from the measured logs.
User repository on GitHub
An adversarial evaluation of TypeSafe's jev decision model: nine experiments and 28 predictions fixed before any data was collected. 123,805 requests, $12.69.
Repo stats from the GitHub API, cached Oct 2026.
An adversarial evaluation of jev, the decision model sold by TypeSafe. It follows a plan written before any request was sent. That plan fixed nine experiments, the sample size of each, and twenty-eight predictions, each stated with the result that would prove it wrong. Nothing reported here was chosen after seeing an outcome. One run produced every number below: 123,805 requests, 138 minutes, $12.69, five failures, all answered by jev-1.13.0.
Quoted from the project's README. Read the full README.
Watch out
Twelve of twenty-five testable predictions held and thirteen were wrong, which the author calls the useful half. Ground truth always comes from a solver or from construction, never from the model, and every number here comes from jev-1.13.0 measured against one plan. Phase 0 is a hard gate and a support-ticket control scored 0.980 throughout, which is what makes the rest readable. Derived figures such as a drift or a rate of decline are rebuilt from the log only for the calibration experiment, the largest remaining gap against the plan. The raw logs are 1.3 GB and published separately compressed with zstd to 47 MB, with a sha256 and record count per file in manifest.json. Python 3.11 and an API key are needed to re-run, and the digest we reviewed ends mid-sentence in the closing section. Copy reviewed against the repository README on 2026-10-04, not from running it.
rorshopping/jev-on-a-laptopPython
erendikmenn/jev-rag-benchmarkPython
idovmamane/dejevuPython
apolinario/decision-indexPython
johnhughes3/LegalForecastBenchPython
mahlernim/jev-korean-benchmarkPythonOriginal write-ups on what this entry is for.
Posts from builders shipping with Jev right now.