Article
CommunityJev Benchmarks & Evaluations
Can Model Answer Probabilities Represent Survey Populations?
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Evaluates Jev option probabilities against 20 binary survey items from five archived waves.
- Compares population-only prediction, prediction given recorded answers, and sequential completion.
- Recorded answers can improve predictive scores while associations remain distorted.
- In one case Jev predicted 16% versus 56% in the survey for a conditional subgroup.
- Reversing question order produced different joint distributions in both tested cases.
Questions about this entry
Short answers drawn from the linked project or page.
It evaluates the option probabilities returned by Jev, a TypeSafe decision model, against linked survey responses, checking predictive scores, answer frequencies, and associations rather than predictive accuracy alone.
Linked responses to 20 binary items drawn from five archived survey waves. Joint comparisons hold the starting population information fixed, and two three-question cases are completed in both question orders.
Three: population-only prediction, prediction given recorded answers, and sequential completion that uses generated answers to predict later responses.
No. Recorded answers can improve predictive scores while associations remain distorted, and large conditional errors also occur.
Among men who described themselves as having traditional values, Jev predicted 16% would see men and women as similar in workplace abilities, compared with 56% in the survey.
Yes. Sequential completion has mixed effects on joint validity, and reversing question order produces different joint distributions in both tested cases.
The evidence concerns selected binary tasks and one model version. The authors support validating the joint events a simulation uses and separating empirical constraints from model assumptions.
More Jev resources like this
SO-Arena is a benchmark for System One models covering 11 domains and five capability families. It measures capability spectrum and decision reliability across 21 models, including decision stability under option-set changes and missing-answer recognition.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Controlled audit of presentation sensitivity in typed decision models, comparing Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces on 9,000 multiply annotated humor, helpfulness, and summarization items.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing S1CogBench, a benchmark that runs 24 cognitive experiments against System One decision models, including Jev and five open models, to test whether typed decisions respond to evidence, feedback, memory load, and future opportunities.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.