Article
CommunityJev Benchmarks & Evaluations
S1CogBench
arXiv paper introducing S1CogBench, a benchmark that runs 24 cognitive experiments against System One decision models, including Jev and five open models, to test whether typed decisions respond to evidence, feedback, memory load, and future opportunities.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Evaluates Jev and five open models on 24 cognitive tasks.
- Described by the authors as the first benchmark of System One decision models on established cognitive experiments.
- Keeps task conditions and scoring separate from disclosed adaptations.
- Reports that decision models vary widely in reliability.
- Finds that speed comes mainly from answering without generation.
Questions about this entry
Short answers drawn from the linked project or page.
S1CogBench is a benchmark that asks what System One decision models, also called typed decision models, can and cannot do on 24 cognitive tasks. It varies task demands and measures how behavior changes, giving a shared test of whether decisions respond to evidence, feedback, memory load, and future opportunities.
The benchmark evaluates five open models plus Jev. It is aimed at System One decision models such as Jev and Laya, which return one fast decision over the legal options rather than generating text.
S1CogBench covers 24 cognitive tasks. The authors keep task conditions and scoring separate from disclosed adaptations when running the evaluations.
The paper reports that decision models vary widely in reliability. It also notes that aggregate scores hide how decisions respond to conditions, so a single summary number can mask behavior changes across task settings.
No. The authors state that acting well does not imply predicting humans, which is one reason cognitive experiments are useful: they measure how behavior changes with task demands rather than only whether a task is completed.
The paper attributes the speed mainly to answering without generation. A typed decision model returns one decision over the legal options and leaves no text to inspect, so there is no generation step to run.
S1CogBench measures behavior on established cognitive experiments, not production routing or guardrail tasks. The authors caution that a fast, well-formed decision does not by itself show the model used the evidence, and that aggregate scores can hide condition-level responses.
More Jev resources like this
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Controlled audit of presentation sensitivity in typed decision models, comparing Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces on 9,000 multiply annotated humor, helpfulness, and summarization items.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper presenting a bias audit protocol for System One decision models, covering BBQ, CrowS-Pairs, and SilverSpoon adapted to the typed interface, twelve models plus a no-thinking ablation, and 1,638,000 decisions with swapped options.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.