Article
CommunityJev Benchmarks & Evaluations
Jev vs GPT-4.1 on a synthetic survey
An independent evaluation asking jev-1.13.0 and GPT-4.1 to stand in for the same 300 Twin-2K-500 respondents across 108 questions: asked as Noul, Jev led GPT-4.1's probabilities on all six measures at $4.02 versus about $136, with no arm predicting individuals.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- jjd-lab
- Last verified
- Oct 2026
Highlights
- Asked as Noul, Jev went 6-0 against GPT-4.1's verbalized probabilities on six measures, at $4.02 billed versus about $136.
- 300 of Twin-2K-500's 2,058 respondents and 108 scored questions; the cheapest persona carries 14 demographic facts.
- Switching the 65 yes/no questions from two-option Choice to Noul moved the distribution gap from 0.1985 to 0.1530 at the same price.
- No arm predicts an individual: per-person correlation sits near zero on 15 of 16 tasks, and humans themselves repeat answers only 81.68% of the time.
- Every arm missed the page's ECE 0.05 bar: Jev Noul 0.1472, Jev Choice 0.2029 and GPT-4.1 0.2393; GPT-4.1's hard-answer arm still won accuracy at 69.32%.
Watch out
Page text fetched 2026-10-17. These are the authors' own measurements from an independent, self-funded evaluation with no TypeSafe or OpenAI involvement, as the page states; Jev is jev-1.13.0 and the GPT-4.1 cost bar prices that arm's token counts at list rates and is not a bill. The 300 respondents are the first 300 rows of the sample - older, whiter and more conservative - and the page warns against reading any absolute number as a population estimate.
Questions about this entry
Short answers drawn from the linked project or page.
Whether a native probability beats a verbalized one: the same 300 survey respondents and 108 questions were answered by Jev, which returns a probability per option as its output type, and by GPT-4.1, which had to fill a probability into an enforced JSON schema.
Twin-2K-500 (Toubia et al., CC BY 4.0): 300 of 2,058 respondents, 108 scored columns, with personas built from 14 demographic facts; a second arm adds 620 of the person's own past answers. Forty of the 108 columns are a pricing task with randomized prices.
Asked as Noul, Jev led GPT-4.1 probabilities on all six measures at $4.02 versus about $136 billed for GPT-4.1 - a thirty-fourth of the cost. Plain GPT-4.1 hard answers still scored two points higher on accuracy but returned no distribution.
It measures how far a model's spread of answers sits from the humans', from 0 to 1. Switching the 65 yes/no questions from a two-option Choice to a Noul moved it from 0.1985 to 0.1530 at the same price, and saturation at probability zero fell from 16.6% of cells to 0.0%.
All arms missed the 0.05 ECE bar the page sets from TypeSafe's 'trained for calibrated decisions' line: Jev Noul 0.1472, Jev Choice 0.2029 and GPT-4.1 0.2393. Jev was the better calibrated by 39% yet still three times over the bar, which the page reads as a statement about demographics-only grounding.
No. Per-person correlation with the human sits near zero on 15 of 16 tasks whatever the persona carries, and adding 620 past answers was not enough to predict individuals. The authors conclude a demographics-grounded twin is usable for a population's distribution, not for one person.
jjd-lab, September 2026, independently and self-funded with no TypeSafe or OpenAI input. Limits: the 300 are the first rows of the sample (older, whiter, more conservative), absolute numbers are not population estimates, and Noul was built after the two-option result so it cannot carry the original verdict.
More Jev resources like this
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
ML Intern Lab's benchmark write-up of open decision models on the LocalLLaMA/typed-decisions test split: 2,000 decisions per model for GEV-26B-Decide, JEV-35B and pplx-decider-v1.1-27b, with accuracy, KL, Brier, ECE and latency tables. The repository states no licence.
Jev Benchmarks & EvaluationsDocs#community#evaluation#benchmark
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Evaluation artifacts for hotchpotch's Bekko System One v0 models on Decision Index 0.2.1: local strict, no-truncation runs scoring 7.37 (17M), 12.83 (68M) and 21.87 (400M), with predictions and environment hashes.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.