Article
CommunityJev Benchmarks & Evaluations
Are AI Decision Models Biased? A Case Study of Jev (arXiv 2610.19619)
A 41,500-response bias audit of Jev across cognitive reflection, five bias paradigms, and a pricing experiment: selected answers look robust, but probabilities move with irrelevant numbers, option order, and framing.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- 41,500 responses across cognitive reflection tasks, five classical cognitive-bias paradigms, and a pricing experiment testing ethnic and gender bias.
- At the level of selected answers, Jev avoids the characteristic choice patterns of several well-documented human biases.
- Its judgments are sensitive to irrelevant numerical information and to the ordering of response options.
- Cognitive-bias manipulations substantially shift returned probabilities even when the selected answer stays the same.
- In 15,000 additional responses across 30 conjunction tasks, Jev violates the same probability rule in 54% of responses.
Watch out
arXiv preprint posted October 14, 2026 (cs.AI, 12 pages). One model on mostly English prompts, and the demographic pricing test is a single scenario, so the small group differences do not establish fairness. The paper's point is that selected-answer scores can overstate reliability: probabilities moved where answers did not.
Questions about this entry
Short answers drawn from the linked project or page.
Three parts: cognitive reflection tasks, five classical cognitive-bias paradigms, and a pricing experiment testing ethnic and gender bias, for 41,500 responses in total, plus 15,000 more responses across 30 conjunction tasks.
Mostly no at the level of selected answers: Jev performs strongly on cognitive reflection and generally avoids the choice patterns associated with several well-documented human biases, with only small demographic-group differences in the pricing scenario.
Sensitivity to irrelevant numerical information and to the ordering of response options, and bias manipulations that shift returned probabilities even when the answer the model selects remains unchanged.
Across 15,000 responses on 30 conjunction tasks, Jev violates the underlying probability rule in 54% of responses, with accuracy varying dramatically across formulations and answer orders, even though it answers a well-known probability problem correctly every time.
Because the probabilities a caller gates on are where the sensitivity shows. Scoring decision models only on familiar benchmarks and selected answers can substantially overstate their reliability in automated decision systems.
Test option order and irrelevant-number variants of your own questions, treat probability outputs as the thing being evaluated rather than the label alone, and re-check prompts when formulations change, since framing moved probabilities in this study.
No, it is an arXiv preprint (cs.AI) posted October 14, 2026, running 12 pages with six figures and three tables. The findings are consistent with earlier option-order and coherence research on the same model class.
More Jev resources like this
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
4GitHub stars
An audit of whether Jev's confidence reflects missing knowledge rather than chance, using over 15 public datasets and 6 generated task families with paired interventions. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An independent reproduction of the shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, finding it uniformly under-confident rather than over-confident. A single fitted temperature removes most miscalibration, while the preregistered isotonic selection failed its held-out contrast.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.