Article
CommunityJev Benchmarks & Evaluations
Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
arXiv study that shuffles candidate options across eight presentations while keeping question and candidate content fixed, testing whether Jev-style System One models resist selection bias. Compares two Jev versions, twelve Jev reproductions, and six generative LLMs on 1,700 questions.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Test shuffles options while keeping question and candidate content fixed, across eight presentations.
- Covers two Jev versions, twelve Jev reproductions, and six generative LLMs.
- Primary collection: 1,700 questions drawn from two benchmarks.
- Jev accuracy varies by less than two percentage points across presentations.
- Cumulative failure set expands by 27-29% relative to the original order.
Questions about this entry
Short answers drawn from the linked project or page.
Whether Jev-style decision structure reduces selection bias. The test shuffles options while keeping question and candidate content fixed, and the paper compares results against fixed-order repetition.
Two Jev versions, twelve Jev reproductions, and six generative LLMs, traced through eight presentations on a primary collection of 1,700 questions from two benchmarks.
Accuracy varies by less than two percentage points, yet the cumulative failure set expands by 27-29% relative to the original order, with first failures continuing into the final presentation.
Aggregation offers limited protection. The paper reports that voting yields modest gains for Jev, and that even unanimous answers are frequently wrong.
No. The paper reports that Jev shows no general advantage over the LLM baselines in preserving correct choices under shuffled presentations.
The loss of initial successes is substantially larger under shuffling than under fixed-order repetition, even for Jev reproductions that are highly repeatable.
Repeatable decisions and decisions that stay correct across equivalent presentations are different properties, so stable accuracy alone cannot establish resistance to selection bias.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing GDGraph, a guideline-derived decision graph framework that compiles radiology guidelines into an executable graph and has a System One model answer localized clinical questions, reporting 90.8% accuracy across 347 cases.
Jev Benchmarks & EvaluationsArticle#arxiv#research#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.