Article
CommunityJev Benchmarks & Evaluations
Can the Confidence and Decisions of System 1 Models Be Trusted?
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Evaluates five open-weight models from 0.8B to 27B against logprob and structured JSON/CoT System 2 baselines.
- Reports provider confidence as a closed-form dispersion transform rather than empirical accuracy.
- Bundling questions in one request caused 11.7% to 22.4% binary flips on Tev1-0.8B, versus 0.1% to 0.2% on Tev1-4B.
- Models at 4B or larger reached ECE_w of 0.030 to 0.087 under syntactic shifts, versus 0.144 logprob and 0.378 JSON.
- S1-SHAP with a logit link recovered 89.3% to 100.0% of disjunctive OR gates, versus 57.1% to 82.1% in probability space.
Questions about this entry
Short answers drawn from the linked project or page.
It evaluates System 1 decision models on calibration under label-preserving perturbations, cross-question interference when multiple questions are bundled in one request, and post-hoc attribution faithfulness, alongside generative System 2 baselines.
Five open-weight models ranging from 0.8B to 27B parameters, compared against generative System 2 baselines using single-token logprob scoring and structured JSON/CoT decoding.
NLI (e-SNLI), adversarial syntactic shortcuts (HANS), synthetic clinical triage (N=900), prompt injection defense (Tensor Trust, N=250), and an 86-case logic-gate trigger suite.
The paper reports it as an exact closed-form dispersion transform: 1 - H(p)/ln K on sub-10B models and a normalized margin on Clef-27B, rather than a measure of empirical accuracy.
The audit found directional cross-question interference, with 11.7% to 22.4% binary flips on Tev1-0.8B compared with 0.1% to 0.2% on Tev1-4B.
On Tensor Trust (N=250), models with 4B parameters or more reached 0.988 to 0.998 AUROC with up to 95.2% recall and 0.0% false alarms at 48 to 60 ms.
The abstract describes an open, deterministic audit benchmark but names no maintainer, license, or repository. It covers five open-weight models from 0.8B to 27B and the listed suites; it reports no results for closed frontier models or production deployments.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Multilingual benchmark of nine suites in 122 languages that evaluates typed decision models such as Jev, Clef, and pplx-decider, and tests whether confidence thresholds fitted on English transfer to other languages.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper comparing two releases of Ru-Jev, a 307M-parameter multilingual cross-encoder trained on thirteen Russian-language tasks, to test whether an augmentation that presents labels in several surface forms helps when callers supply their own option wording.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Controlled audit of presentation sensitivity in typed decision models, comparing Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces on 9,000 multiply annotated humor, helpfulness, and summarization items.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing S1CogBench, a benchmark that runs 24 cognitive experiments against System One decision models, including Jev and five open models, to test whether typed decisions respond to evidence, feedback, memory load, and future opportunities.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.