CommunityJev Benchmarks & Evaluations
Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges (arXiv 2610.18122)
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Runs the CALM benchmark's bias tests on eight typed decision models and seven LLM judges from 2024 and 2026.
- Every TDM is biased, failing three to eight of twelve tests, with emotional tone affecting them most.
- Rewriting the better answer in an emotional tone lowers a typical TDM's accuracy from 99% to 42%.
- All eight TDMs change verdicts when the answers swap places, which the 2026 LLM judges hardly do.
- TDMs are more consistent and 37 times cheaper than 2026 LLM judges, but good calibration does not protect against bias.
Watch out
arXiv preprint posted October 13, 2026 (cs.CL) by Fabian Lukassen, Christoph Weisser, and Alexander Silbersdorff. The findings rest on the CALM benchmark's bias tests, and the authors note both kinds of judge are tripped up by the same prompts; no specific deployment of a TDM judge is validated.
Questions about this entry
Short answers drawn from the linked project or page.
Eight typed decision models (TDMs) and seven LLM judges from 2024 and 2026, all tested with the CALM benchmark's bias tests.
Yes. Every TDM is biased in three to eight of the twelve tests, and the authors report TDMs are more biased than the 2026 LLM judges.
Emotional tone affects TDMs most. Rewriting the better answer in such a tone lowers the accuracy of a typical TDM from 99% to 42%.
All eight change verdicts when the answers swap places, a behavior the 2026 LLM judges hardly show. Several also side with an answer most people allegedly prefer or that a famous model allegedly wrote.
No. The TDM probabilities show most tests mainly flip verdicts the TDM was unsure of, and the paper states good calibration does not protect against bias.
The authors report TDMs are more consistent and 37 times cheaper than 2026 LLM judges, which makes them attractive judges if their biases are tested for the task.
The results suggest TDMs can be attractive judges when their biases are tested for the task at hand, since both kinds of judge are tripped up by the same prompts.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An evaluation of JEV-as-a-Judge, a decision-only judge that returns label probabilities and uses its confidence to accept a verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges it comes within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
An independent reproduction of the shipped Laya Typed-Decisions checkpoint, a 421M-parameter ModernBERT-large assessor, finding it uniformly under-confident rather than over-confident. A single fitted temperature removes most miscalibration, while the preregistered isotonic selection failed its held-out contrast.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A comparison of JEV with LLMs and human coders on seven political science replications. JEV matches or approaches GPT-6 Luna and Qwen3.8-27B on many tasks, but the authors find no cost advantage over GPT-6 Luna at OpenAI's batch prices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
Training recipes, looped architectures, and language ports: what the open side of the typed-decision class shipped this month, and how to pick by runtime and calibration instead of the leaderboard.
Jev Models & Reimplementations#open-models#training#local
Read article
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.