Article
CommunityJev Benchmarks & Evaluations
System One Models Are Not Better Rerankers
An arXiv study comparing decision fine-tuning against reranker fine-tuning across six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, concluding that decision fine-tuning never produces the better reranker.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Compares untuned zero-shot models, decision fine-tunes, and reranker fine-tunes on six Qwen backbones.
- Reranker fine-tuning wins from 0.6B to 9B by 5 to 29 points nDCG@10 on every backbone.
- At 27B parameters the two fine-tuning approaches tie.
- Calibration advantage of decision models holds only for certain question phrasings, from 4B up.
- Up to 9B, decision fine-tuning adds nothing over asking the untuned model.
Questions about this entry
Short answers drawn from the linked project or page.
It tests whether the decision fine-tuning behind System One models makes a better reranker than plain reranker fine-tuning of the same backbone, comparing untuned zero-shot models, decision fine-tunes, and reranker fine-tunes.
Six Qwen backbones ranging from 0.6B to 27B parameters, evaluated on ten TREC DL and BEIR collections, including zero-shot collections.
The authors conclude that decision fine-tuning never makes the better reranker: from 0.6B to 9B reranker fine-tuning wins on every backbone, and at 27B the two approaches tie.
From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, including on zero-shot collections.
No. Asked whether a passage answers the query, decision models are better calibrated than published reranker fine-tunes from 4B on, but asked a slightly different question from prior work they lose that advantage.
No. Up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold, so calibration does not make up for a worse ranking.
They recommend fine-tuning a reranker rather than relying on decision fine-tuning, since decision fine-tuning adds nothing over the untuned model up to 9B.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.