CommunityJev Benchmarks & Evaluations
JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications (arXiv 2610.06625)
A comparison of JEV with LLMs and human coders on seven political science replications. JEV matches or approaches GPT-6 Luna and Qwen3.8-27B on many tasks, but the authors find no cost advantage over GPT-6 Luna at OpenAI's batch prices.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Compares JEV with published human coders, GPT-6 Luna, and open-weight Qwen3.8-27B across seven political science replications.
- JEV matches or comes close to both LLMs across a variety of text annotation tasks.
- No cost advantage over GPT-6 Luna at OpenAI's batch prices, against JEV's advertised cost and speed advantage.
- Asked once per question, JEV's probabilities are better calibrated than GPT-6 Luna's token probabilities.
- JEV's calibration is not consistently better than Qwen3.8-27B's; ease of parsing probabilities is its obvious remaining advantage.
Watch out
arXiv preprint posted October 5, 2026 and updated October 6, 2026 (cs.CL, cs.CY; 71 pages) by Steven Denney and Matthew DiGiuseppe. Results are limited to seven political science replications and the named comparison models, and the cost findings depend on OpenAI's batch pricing at the time of the study.
Questions about this entry
Short answers drawn from the linked project or page.
JEV, a current mid-tier commercial LLM (GPT-6 Luna), an open-weight alternative (Qwen3.8-27B), and human coders from published research, across seven political science replications.
The authors find JEV matches, or comes close to, both LLMs in a variety of tasks, which they say establishes its suitability for social science text annotation and scaling.
No. The paper reports no cost advantage over GPT-6 Luna at OpenAI's batch prices, despite JEV being advertised with a dramatic cost and speed advantage.
When each question is asked once, JEV's probabilities are better calibrated than GPT-6 Luna's token probabilities but not consistently better than Qwen3.8-27B's.
Unless researchers have a need for speed, its only obvious advantage is the ease of parsing the underlying choice probabilities.
Text annotation and scaling tasks drawn from seven replications in political science, compared against human coders and two LLMs.
Social scientists who want to annotate or scale large corpora of text quickly and need a reliable indicator of classifier uncertainty, the audience the paper addresses.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: 100,000 viral posts in 20.4 seconds
X: SEO and GEO fixes, 90% cheaper
X: Hard-coded rules moved to Jev
YouTube: Joe Maddalone tries Jev as a routing and filtering step, with a transcript-analyzer demo.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A benchmark providing exact instance-level class probabilities across three simulated text domains: peer review, covert information transfer, and credit risk. Because the data-generating process is defined, exact probabilities and Bayes error are computable, enabling KL-divergence scoring of reported probabilities.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.