A bias audit is not an accuracy test with a different name. It asks whether the same substantive case gets a different judgment when an irrelevant attribute changes: a persona, a language, an answer's position, or the emotional tone of the text. The most useful audits separate three layers (who the model is asked to be, what language the question is in, and how the options are framed) because the fixes differ.
The other distinction that matters is between the selected answer and the probability behind it. Several audits find the top answer stable while the distribution moves. If your gate reads the probability, that movement is the finding.
What a bias audit tests
A controlled audit changes one attribute at a time and holds the evidence fixed. That is harder than it sounds, because personas change the wording, translations change the item, and position swaps change the order. The bias case study of Jev is a good model of the design: it runs cognitive reflection tasks, five classical bias paradigms, and a pricing experiment testing ethnic and gender bias, for 41,500 responses, plus 15,000 more across 30 conjunction tasks.
| Audit layer | What changes | What it can show | What it cannot show |
|---|---|---|---|
| Persona | Stated nationality, age, gender | Whether profiles shift with identity cues | Whether deployment traffic behaves the same |
| Language | Item language, persona language | Whether the effect is item- or description-driven | Fluency outside the tested languages |
| Framing | Option order, tone, irrelevant numbers | Whether probabilities are sensitive to presentation | Whether the top answer is wrong |
| Judge position | Which answer appears first | Position bias in pairwise verdicts | Which answer is better |
Personas and cultural values
Calibrated to Whom put 288,000 answers to the 24 Values Survey Module 2013 items through Jev, using 12 matched Saudi and 12 matched American personas, no persona, English and Arabic, under eight request formulations. Answers were highly repeatable, with an ICC of 0.997. Without a persona, they resembled Jev's own American personas.
A Saudi persona moved answers toward the human Saudi–US difference, reproducing 87% of its size in English but 62% in Arabic. A language cross showed the smaller Arabic difference came from the language of the items, not the language of the persona description. Age shifted profiles about as much as nationality, gender shifted them more for Saudi than for American personas, and Jev was less confident in Arabic and for Saudi personas.
The caveats bound the claim: one decision-only model, one values instrument, survey-style prompts rather than deployment behavior, and long-term orientation reversing in Arabic. The finding is not that the model has a fixed cultural profile. It is that the profile is a function of persona and item language, and the two do not move together.
Selected answers versus probabilities
The bias case study found that, at the level of selected answers, Jev largely avoided the characteristic choice patterns of several well-documented human biases. It performed strongly on cognitive reflection and showed only small demographic-group differences in the single pricing scenario. That is the reassuring half.
The other half is that its judgments were sensitive to irrelevant numerical information and to the ordering of response options. Cognitive-bias manipulations substantially shifted returned probabilities even when the selected answer stayed the same. Across 15,000 responses on 30 conjunction tasks, Jev violated the underlying probability rule in 54% of responses, with accuracy varying dramatically across formulations and answer orders, even though it answered a well-known probability problem correctly every time.
The paper's warning follows directly: scoring decision models only on familiar benchmarks and selected answers can substantially overstate their reliability in automated decision systems. The probabilities a caller gates on are where the sensitivity shows. The jaggedness notes add that the same question can disagree across primitives.
Language and replication
JEV versus LLMs compared Jev with published human coders, GPT-6 Luna and Qwen3.8-27B across seven political science replications. Jev matched or came close to both LLMs across a variety of text annotation tasks. It found no cost advantage over GPT-6 Luna at OpenAI's batch prices, against Jev's advertised cost and speed advantage. Asked once per question, Jev's probabilities were better calibrated than GPT-6 Luna's token probabilities, but not consistently better than Qwen3.8-27B's. The authors call ease of parsing probabilities its obvious remaining advantage unless speed matters.
That is a narrower claim than "Jev is unbiased across languages," and it is more useful. On these seven replications, the accuracy gap was small; the cost story depended on the comparison price; the calibration advantage was real against one model and not the other. The study covers seven replications and two comparison models, and its cost finding depends on batch pricing at the time, so it is evidence to replicate rather than a general law.
The language result from the persona audit points the same way. The item language, not just the persona description, changes the profile. A multilingual deployment needs per-language validation, not a translated prompt and a shared threshold.
Judges: tone and position
Typed Decision Models as Judges ran the CALM benchmark's bias tests on eight typed decision models and seven LLM judges from 2024 and 2026. Every TDM was biased in three to eight of the twelve tests. Emotional tone affected them most: rewriting the better answer in an emotional tone lowered a typical TDM's accuracy from 99% to 42%. All eight changed verdicts when the answers swapped places, behavior the 2026 LLM judges hardly showed. Several also sided with an answer most people allegedly prefer or that a famous model allegedly wrote. TDMs were more consistent and 37 times cheaper than the 2026 LLM judges, but good calibration did not protect against bias.
Same Attack, Different Judge adds the adversarial layer. Under one policy and identical payloads, eight forgeries of Jev's structured input bypassed none of its moderation decisions, while a sentence asking only why someone might make a harmful request bypassed 9% of its blocks, against 56% for GPT-6 Astra. On phishing, forgeries bypassed 19–29% of the flagged emails, and unattacked Jev detected 62% of phishing emails against 77–81% for the GLM judges. Jev was the fastest judge but also flagged the most benign requests, at 27%.
Jev-ToM tests a different capability. It compares five typed decision models with seven generative baselines on 11,117 English questions from nine Theory of Mind benchmarks across six capability groups. Jev reached 73.2% source-balanced accuracy and ranked first or second across all six groups. Related-question cues accounted for most of its downstream gains. Confidence-based selection remained benchmark-dependent, and supplied reasons could repair errors or overturn correct answers. The results cover benchmark questions, not live assistant deployments. The confidence guide is the reminder that confidence is a derived statistic, not a second opinion.
What an audit can and cannot establish
- Test probabilities, not only selected answers.
- Hold the item fixed when changing persona or language; otherwise you measure the translation.
- Separate item language from persona-description language.
- Swap answer positions and rewrite tone before trusting a judge.
- Report group differences with intervals and a base rate; a single pricing scenario is not a fairness result.
- Use human adjudication where the label is contested.
- Re-run the audit on your own population, language mix and document types.
The practical implications are in multilingual decision design and guardrails and prompt injection. For the calibration side of the same problem, see what the papers found. The sycophancy, language support and calibration entries cover the underlying terms.
