CommunityJev Benchmarks & Evaluations
Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV (arXiv 2609.36399)
An audit of TypeSafe's JEV with the Values Survey Module 2013: 24 items asked as 12 matched Saudi and 12 matched American personas, without a persona, in English and Arabic, under eight request formulations, for 288,000 answers.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Puts 288,000 answers to the 24 Values Survey Module 2013 items through JEV.
- Answers were highly repeatable (ICC 0.997), and without a persona they resembled its own American personas.
- A Saudi persona moved answers toward the human Saudi–US difference, reproducing 87% of its size in English but 62% in Arabic.
- The smaller Arabic difference comes from the language of the items, not the language of the persona description.
- Age shifted profiles about as much as nationality, and JEV was less confident in Arabic and for Saudi personas.
Watch out
arXiv preprint posted September 28, 2026 and updated October 3, 2026 (cs.CL, cs.AI) by Bushra Asseri and Abdulaziz Asseri. It covers one decision-only model on one values instrument, with long-term orientation reversing in Arabic, and the findings concern survey-style prompts rather than deployment behavior.
Questions about this entry
Short answers drawn from the linked project or page.
The Values Survey Module 2013: 24 items asked as 12 matched Saudi and 12 matched American personas and without a persona, in English and Arabic, under eight ways of formulating the request.
288,000 answers across the persona, language, and request-formulation combinations.
They were highly repeatable, with an ICC of 0.997, and without a persona they resembled those of its own American personas.
When the persona was Saudi rather than American, answers moved in the direction of the human Saudi–US difference, reproducing 87% of its size in English but 62% in Arabic, with long-term orientation reversed.
A language cross shows it comes from the language of the items, not the language of the persona description.
Age shifted the profiles about as much as nationality, gender shifted them more for Saudi than for American personas, and JEV was less confident in Arabic and for Saudi personas.
No. The patterns held in every request design, even though the model never generates text.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A comparison of JEV with LLMs and human coders on seven political science replications. JEV matches or approaches GPT-6 Luna and Qwen3.8-27B on many tasks, but the authors find no cost advantage over GPT-6 Luna at OpenAI's batch prices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.