Article
CommunityJev Benchmarks & Evaluations
XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Nine suites covering 122 languages, with a twelve-language suite used for the main model comparison.
- Three open models fine-tuned for typed decisions beat their Qwen base models in every language on the twelve-language suite.
- Accuracy gains over local base models were larger in languages with less web text than in those with more.
- A confidence threshold fitted on English for 5% errors gave a higher error rate on XNLI, including German and French.
- On Belebele, Decision-1 is best in English but falls far behind Jev in Burmese.
Questions about this entry
Short answers drawn from the linked project or page.
A multilingual benchmark of nine suites covering 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, which return a probability for each answer an application provides.
Nine suites in 122 languages, with a twelve-language suite used for the main model comparison.
On the twelve-language suite, three open models fine-tuned for typed decisions beat their Qwen base models in every language.
On new test data with local base models, the accuracy gain was larger in languages with less web text than in those with more, though the models fixed a similar share of base-model errors in both.
No. The best model on the twelve-language suite is best on only two other suites, and the best English model on each suite, chosen in hindsight, is significantly beaten in about a quarter of suite and language combinations.
A confidence threshold fitted on English to allow 5% errors gives a higher error rate on XNLI, even in German and French.
Results from one suite or from English are a poor guide to how a model and its threshold behave in other languages; for example, Decision-1 is best in English on Belebele but falls far behind Jev in Burmese.
More Jev resources like this
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
SO-Arena is a benchmark for System One models covering 11 domains and five capability families. It measures capability spectrum and decision reliability across 21 models, including decision stability under option-set changes and missing-answer recognition.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Controlled audit of presentation sensitivity in typed decision models, comparing Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces on 9,000 multiply annotated humor, helpfulness, and summarization items.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing S1CogBench, a benchmark that runs 24 cognitive experiments against System One decision models, including Jev and five open models, to test whether typed decisions respond to evidence, feedback, memory load, and future opportunities.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.