User repository on GitHub
Measured: asking TypeSafe Jev N questions in one call bills the state once. 2,976 real requests, raw data, exact billing check.
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
CommunityJev Benchmarks & Evaluations
Jev Fanout Bench
Reproducible billing and stability measurement of Jev's fan-out behavior across 2,976 requests to jev-1.13-20260917: asking N questions in one call bills the state once, with raw data, exact invoice checks and a batched-versus-single answer comparison.
- Stars
- 0
- Language
- Python
- License
- MIT
- Published by
- Community
- Last verified
- Oct 2026
About this project
A reproducible measurement of one claim about Jev, TypeSafe AI's System One decision model: If that holds on the bill, asking N questions in one request should cost the state once, and asking them in N requests should cost it N times. TypeSafe's own parallel-questions cookbook measured this once, on one document, as 12.2x cheaper. This repository measures it across state sizes and question counts, and checks three things the cookbook does not:
Quoted from the project's README. Read the full README.
Works forJev for Evaluation
Used inJev for Software & DevTools
- Languages
- Python
- Platforms
- OpenRouterCloudflare
- Capabilities
- Open sourceRequires API key
Highlights
- 2,976 requests totaling $0.156; usage.cost matched input_tokens x $0.042/1M to the last decimal and 122,844 output tokens were not charged
- Every request carries about 261 input tokens of fixed overhead; a one-word state bills F = 261
- Median input-token savings at eight questions: 76% tiny control, 77% small, 84% medium, 86% large state
- Batched-versus-single answer differences matched the repeat-request noise floor; the top Choice option never differed
- Round 2 found context limits bracketing 32,768 content tokens and 65,536 total, and 1,000 questions succeeded in one call
Watch out
MIT for code, CC0 for inputs and results. Unofficial and independent; the headline run is from 2026-09-23 and TypeSafe had paused new signups, so newer limits or pricing may differ.
Questions about this entry
Short answers drawn from the linked project or page.
Sending N questions in a single Jev request versus N separate requests. Jev evaluates every question against one state, so batching should bill that state once, and the measurement checks the bill.
Yes. The same requests run through OpenRouter's System One-compatible endpoint or Cloudflare Workers AI with their keys, and --dry-run exercises the pipeline with no key or network.
Python 3.10+ and the standard library only. python bench.py run sends 2,976 requests and about 2 million input tokens in roughly 15 minutes, around $0.09 at the published $0.042 per million input tokens with output free, then python bench.py report writes the summary and the fanout chart. It pins jev-1.13.0 rather than jev-latest, which can move mid-study.
Not beyond the noise floor. Every question was asked both inside the batch and alone, and every request was repeated, so the batched-versus-single difference equals the difference between two identical requests: Noul 0.005 mean against 0.005 repeated, Score 0.010 against 0.010, and the top Choice option never differed.
About 261 input tokens of fixed overhead per request, readable on its own because a one-word state still bills 261. On top of that a question costs roughly 8 tokens of framing, about 1 per English instruction word, about 8 per Choice option or 21 with a six-word description, and 8 per Score level, all linear at R-squared 0.999 or better. Output tokens are not charged.
Bisection brackets content at 32,768 tokens and total at 65,536, and the overhead counts toward neither. There is no question-count cap: 1,000 questions in one call succeeded. Rate limits for jev-1.13.0 moved on 2026-09-30 from 250,000 tokens per second and 1,200 requests per minute to 100,000 and 40 per second, with the crossover at 2,500 billed tokens per call.
Keep questions in English. Translating the ticket moved yes/no answers about 3-5 times the English repeat noise, translating the questions as well moved them further in all seven languages and cost 9-48% more tokens. Reordering Choice options never changed the top option, but dropping descriptions or paraphrasing moved probabilities by 2-10 points.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: New signups no longer get free credits.
YouTube: Gary Explains the catch: Jev understands natural language but answers with structured values and confidence.
YouTube: David Ondrej explains the architecture behind Jev and how to build a business on it.
More Jev resources like this
0GitHub stars
Small test harness and benchmark for Jev through OpenRouter's Decisions API. It ships a thin decide wrapper, 17 documentation-derived cases with 23 checks covering Noul, Choice and Score, and a script comparing Jev against GPT-5.6 Terra and Claude Opus 5.
Jev Benchmarks & Evaluations#community#benchmark#openrouter
Community
souvikr/jev-testPython
souvikr/jev-testPython0GitHub stars
Head-to-head benchmark of fast typed classifiers, TypeSafe Jev, OpenAI Decisions and Cloudflare Clef, against Claude models on a frozen, labeled corpus of 2,321 cases across 31 tasks, measuring accuracy, calibration, p95/p99 latency and registry-priced cost.
Jev Benchmarks & Evaluations#community#benchmarking#calibration
Community
CMaintz/classifier-benchPython
CMaintz/classifier-benchPython15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython2GitHub stars
Pre-registered benchmark adding Jev 1.13 to Stanford MedHELM's MedHallu results: one Noul asks whether a proposed answer misrepresents its PubMed abstract. Jev scored 92.9% on 1,000 test items at a 204 ms median and USD 0.03 per 1,000 checks through OpenRouter.
Jev Benchmarks & Evaluations#community#medical#hallucination
Community
stperic/jev-medhallu-benchmarkPython
stperic/jev-medhallu-benchmarkPython2GitHub stars
Independent evaluation running Jev jev-1.13.0 and GPT-4.1 as the same 300 Twin-2K-500 synthetic survey respondents across 24,596 paired cells, finding that asking a yes/no item as a Noul rather than a Choice moved results more than the model gap, at a thirty-fourth of the cost.
Jev Benchmarks & Evaluations#community#benchmark#calibration
Community
jjd-lab/jev-synthetic-surveyPython
jjd-lab/jev-synthetic-surveyPython3GitHub stars
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonFrom the Jev community
Posts from builders shipping with Jev right now.