CommunityJev Benchmarks & Evaluations
Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks (arXiv 2610.11978)
A comparison of Jev with 19 LLMs across 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding. Jev is competitive with frontier LLMs on knowledge and commonsense tasks, but on MathQA it trails the frontier median by 17.7 points.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Evaluates Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding.
- Compares it with 19 LLMs in three tiers: frontier, representative, and small.
- Jev obtains the best score on MMLU-Redux and ARC-Challenge.
- Outside mathematics it outperforms most representative LLMs and all small LLMs.
- On MathQA it is 17.7 points below the frontier median and scores lower than all 19 LLMs.
Watch out
arXiv preprint posted October 8, 2026 (cs.CL; 10 pages) by Xing Li and co-authors. It is a benchmark-only evaluation against the named LLM tiers, the abstract reports no calibration or cost analysis, and no code release is mentioned.
Questions about this entry
Short answers drawn from the linked project or page.
Jev is evaluated on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compared with 19 LLMs in three tiers: frontier, representative, and small.
It is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge.
Outside mathematics, Jev outperforms most representative LLMs and all small LLMs.
On mathematical word problems: on MathQA it is 17.7 points below the frontier median and scores lower than all 19 LLMs.
They can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation.
Multiple-choice benchmarks spanning knowledge, reasoning, and multilingual understanding, including MMLU-Redux, ARC-Challenge, and MathQA.
Teams choosing between a specialized decision model and a general-purpose LLM for structured multiple-choice tasks, since it maps which task types transfer well.
More Jev resources like this
A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
VictorYXL/GraphDecisionBenchPython
VictorYXL/GraphDecisionBenchPythonA pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces JEVal, a bilingual benchmark of 11,257 instances from 36 datasets across 10 application domains, and evaluates 25 model configurations spanning general decision models and generative LLMs. It also proposes SII-Decision-4B and 27B, distilled models answering a typical query in about 0.1 s.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev community
Posts from builders shipping with Jev right now.