Article
CommunityJev Benchmarks & Evaluations
Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects
arXiv paper evaluating 67 systems, including 47 decision models and 20 LLMs, on 43 Arabic tasks across 9 task families under a shared zero-shot protocol. Jev 1.14.0 scores 69.24 against 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Evaluates 67 systems, 47 decision models and 20 LLMs, on 43 tasks across 9 task families.
- Jev 1.14.0 reaches a mean task-primary metric of 69.24, versus 68.77 for Jev 1.13.0.
- Gemma-4-31B scores 71.81, the highest on this aggregate.
- Pooled dialect-minus-MSA accuracy gaps for Jev 1.14.0 range from -7.94 to -0.35 percentage points across four datasets.
- Removing recorded English controls, repeated cross-task items, or both leaves the full system order unchanged.
Questions about this entry
Short answers drawn from the linked project or page.
It compares 67 systems, 47 decision models and 20 LLMs, on 43 Arabic tasks across 9 task families using a shared zero-shot protocol, and examines how performance shifts between Modern Standard Arabic and dialects.
Jev 1.14.0 records a mean task-primary metric of 69.24 on a scale multiplied by 100, against 68.77 for Jev 1.13.0. Gemma-4-31B scores 71.81, the highest on this aggregate.
In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets. Dialect-specific results vary more, and SIB-200's source-clustered interval includes zero.
No. The paper reports that removing the recorded English controls, repeated cross-task items, or both leaves the full system order unchanged.
All systems are run under a shared zero-shot protocol over an Arabic-focused frozen suite that includes English controls in three parallel resources, which are also removed in a separate descriptive comparison.
No. The authors state the findings are descriptive and do not establish a general model ranking for dialect robustness, nor do they explain the causes of the observed gaps.
It is posted on arXiv as 2611.10802, titled Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects.
More Jev resources like this
arXiv paper presenting AbductionBench, a unified framework and taxonomy for abductive reasoning tasks, instantiated on 27 datasets across ten domains and used to evaluate eight LLMs from four model families on single-choice explanation selection.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Same Facts, Different Remedy: Linguistic Profiling and Arbitrariness in LLM and System One Decisions
arXiv paper auditing three remedy-decision systems on US consumer financial complaints: GPT-4o-mini, GPT-5-mini, and Jev, a probabilistic System One model returning a distribution over remedies. It separates group-scale disparities from individual decision instability.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper auditing AI-text humanizers for fact retention and detector evasion, using a rule-based checker and a zero-shot Jev question to separate human originals from humanized rewrites.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting RouteAlign, a method that uses task requirements to structure model performance measurement and LLM service selection. Reports quality gains over open Jev-style decision backends at lower generation cost.
Jev Benchmarks & EvaluationsArticle#arxiv#routing#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.