Article
CommunityJev Benchmarks & Evaluations
Is Jev a Good Multilingual Judge?
arXiv paper testing whether Jev, a decision model that outputs judgment probabilities, works as a multilingual judge. It compares Jev with generative LLM judges across four annotated benchmarks, three evaluation formats, and up to 47 language varieties.
- Format
- Article
- Reading time
- 1 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Evaluates Jev against generative LLM judges on four multilingual benchmarks with human annotations.
- Covers up to 47 language varieties and three formats: binary decisions, pairwise comparisons, and graded ratings.
- Reports Jev at approximately 1/31 to 1/61 of GPT-6.1's evaluation cost.
- Jev is competitive on most multilingual judging tasks but falls behind GPT-6.1 on knowledge-intensive answer verification.
- Finds Jev more likely to accept incorrect answers in low-resource languages than in English.
Questions about this entry
Short answers drawn from the linked project or page.
It evaluates Jev as a multilingual judge against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties.
Three formats: binary decisions, pairwise comparisons, and graded ratings.
The paper reports Jev running at approximately 1/31 to 1/61 of GPT-6.1's evaluation cost.
On knowledge-intensive answer verification, where GPT-6.1 outperforms Jev even though Jev is competitive on most other multilingual judging tasks.
Jev is more likely to accept incorrect answers in low-resource languages than in English.
The paper reports that Jev's probability outputs help identify errors across the judging tasks studied.
Recommendations on evaluation instructions, confidence-based selection, and model routing for using Jev as a multilingual judge.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
X: A group chat bot with decision room routing
More Jev resources like this
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
SO-Arena is a benchmark for System One models covering 11 domains and five capability families. It measures capability spectrum and decision reliability across 21 models, including decision stability under option-set changes and missing-answer recognition.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
arXiv paper introducing hybrid decision models (HDM) with a self-routing block that estimates the value of thinking. Evaluates on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77, and MiniWoB; on JevBench HDM-4B scores 0.857 vs Jev's 0.851 at median 0.179s vs 0.652s.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper examining numerical nondeterminism in vLLM Split-K LoRA kernels and introducing a deterministic Triton Split-K implementation, measured on GSM8K, MBPP, and Jev-style structured toxicity decisions over Civil Comments.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.