Article
CommunityJev Benchmarks & Evaluations
SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models
SO-Arena is a benchmark for System One models covering 11 domains and five capability families. It measures capability spectrum and decision reliability across 21 models, including decision stability under option-set changes and missing-answer recognition.
- Format
- Article
- Reading time
- 1 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Covers 11 domains and five capability families.
- Evaluates 21 models on capability spectrum and decision reliability.
- Measures decision stability under controlled option-set changes.
- Measures missing-answer recognition when no valid option is available.
- The best-performing System One model ranked only seventh in reliability.
Questions about this entry
Short answers drawn from the linked project or page.
It assesses decision reliability through two complementary measures: decision stability under controlled option-set changes, and missing-answer recognition when no valid option is available. Accuracy alone is treated as insufficient for understanding System One behavior.
SO-Arena characterizes System One models across 11 domains and five capability families.
Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains.
It is the ability to recognize that no valid option is available, tested as one of two complementary reliability measures alongside decision stability under controlled option-set changes.
Yes. Reasoning-intensive tasks in the benchmark further enable system one-two comparison, alongside the capability spectrum and reliability measurements.
Strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability.
The material describes SO-Arena as a benchmark introduced in an arXiv preprint; it does not state whether a hosted leaderboard or public submission service is available.
More Jev resources like this
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Controlled audit of presentation sensitivity in typed decision models, comparing Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces on 9,000 multiply annotated humor, helpfulness, and summarization items.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing S1CogBench, a benchmark that runs 24 cognitive experiments against System One decision models, including Jev and five open models, to test whether typed decisions respond to evidence, feedback, memory load, and future opportunities.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.