Article
CommunityJev Benchmarks & Evaluations
When the Top Answer Decides: Gender Bias in Decision Models
arXiv paper measuring gender bias in decision models, testing TypeSafe's closed Jev and five open models on gender-swapped yes/no decisions, BBQ, StereoSet, and Fulgu and Capraro's gender-bias studies, with results on top-answer flips and confidence-gate effects.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Tests TypeSafe's closed Jev and five open decision models on gender-swapped yes/no decisions about people.
- Five of the six models give women significantly more favourable answers than men, +0.05 to +0.10 in log-odds.
- A gender swap changes 1-3.5% of top answers; Jev's changes partly through its call-to-call variation.
- The recommended confidence gate raises the stereotyped share of kept answers in ambiguous BBQ gender questions.
- All six models rate violence for a greater good as more acceptable against a man than against a woman.
Questions about this entry
Short answers drawn from the linked project or page.
It tests TypeSafe's closed Jev alongside five open decision models. Decision models are described as language models built to route, classify and triage, returning option probabilities from one forward pass that an application then acts on.
The evaluation uses gender-swapped yes/no decisions about people from Tamkin et al. (2023), BBQ, StereoSet, and the gender-bias studies of Fulgu and Capraro (2024).
Five of the six models give women significantly more favourable answers than men, between +0.05 and +0.10 in log-odds. A gender swap changes 1-3.5% of top answers, and for Jev this happens partly through its call-to-call variation.
The gate removes many biased commitments, but it raises the stereotyped share of the answers it keeps on ambiguous BBQ gender questions: granite-switch moves from 0.76 to 0.85 and pplx from 0.81 to 1.00.
Probes show preferences in the same models. All six rate violence for a greater good as more acceptable against a man than against a woman, and two default to girl when forced to guess a child's gender without a cue.
The paper reports that the two kinds of measure differ, and argues audits should read a model the way its application does, since the application acts on the top answer rather than on the full probability distribution.
The authors state they found no bias measurement in the cards of the decision models they examined, which are offered off the shelf and trained for the decision format rather than for fairness or any one application.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study evaluating Jev as a decision reader of truncated LLM reasoning across seven reasoners and five multiple-choice benchmarks, comparing accuracy and estimated cost against forced answering at budgets from 500 to 16,000 tokens.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.