Article
CommunityJev Benchmarks & Evaluations
Same Facts, Different Remedy: Linguistic Profiling and Arbitrariness in LLM and System One Decisions
arXiv paper auditing three remedy-decision systems on US consumer financial complaints: GPT-4o-mini, GPT-5-mini, and Jev, a probabilistic System One model returning a distribution over remedies. It separates group-scale disparities from individual decision instability.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Audits GPT-4o-mini, GPT-5-mini, and Jev on sociolinguistic rewrites of US consumer financial complaints.
- Monetary-remedy gaps by age, education, and SES are within 0.5 percentage points.
- Two style versions of the same complaint get different monetary decisions 2.6-6.3% of the time.
- All three systems respond 49-387 times more strongly to remedy-relevant facts than to style.
- Standardizing style cut GPT-4o-mini's education and SES gaps by 94% and 87% but reversed two Jev gaps.
Questions about this entry
Short answers drawn from the linked project or page.
It audits three remedy-decision systems using sociolinguistic rewrites of US consumer financial complaints that are instructed to preserve every fact, separating group-scale disparities from individual decision instability.
Two generative LLMs returning a single label, GPT-4o-mini and GPT-5-mini, and a probabilistic System One model returning a distribution over remedies, Jev.
Jev is described as a probabilistic System One model that returns a distribution over remedies rather than a single label.
Older, college-educated, and higher-SES styles received slightly more generous remedies, but every monetary-remedy gap was within 0.5 percentage points.
Two style versions of the same complaint received different monetary decisions 2.6-6.3% of the time.
It reduced GPT-4o-mini's education and SES gaps by 94% and 87% while retaining fact sensitivity, but reversed two of the probabilistic model's gaps and did not make individual decisions more stable.
They recommend that fairness audits of LLM decision systems report group disparities and individual instability as separate outcomes.
More Jev resources like this
arXiv paper auditing AI-text humanizers for fact retention and detector evasion, using a rule-based checker and a zero-shot Jev question to separate human originals from humanized rewrites.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Empirical study evaluating four specialized decision models on zero-shot text classification in Vietnamese across 13 classification settings and six task groups, testing English, Vietnamese, and bilingual label descriptions plus teencode, typo, and diacritic perturbations.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#classification
Community
arXiv paper presenting AbductionBench, a unified framework and taxonomy for abductive reasoning tasks, instantiated on 27 datasets across ten domains and used to evaluate eight LLMs from four model families on single-choice explanation selection.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper evaluating 67 systems, including 47 decision models and 20 LLMs, on 43 Arabic tasks across 9 task families under a shared zero-shot protocol. Jev 1.14.0 scores 69.24 against 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.