Article
CommunityJev Benchmarks & Evaluations
Decision and System One models in rheumatology: accuracy and calibration
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Evaluated Jev 1.13 on 172 Spanish specialty-training rheumatology questions in Spanish and English.
- Jev scored 93.0% in Spanish and 91.9% in English on the expanded RheumaMIR dataset.
- On 70 RheumBench vignettes Jev reached 50.0% accuracy against 9.6% chance.
- Jev answers at probability >=0.90 were 98.5% correct on RheumaMIR but 84.6% on vignettes.
- Jev showed overconfidence on vignettes (mean probability 0.659; over/underconfidence index +0.159).
Questions about this entry
Short answers drawn from the linked project or page.
Two benchmarks: 172 rheumatology questions from Spanish specialty-training examinations (an expanded RheumaMIR dataset) in Spanish and English, and 70 English diagnostic vignettes from RheumBench with 5-18 candidate diagnoses, 52 of which included the reference diagnosis.
Jev's accuracy was 93.0% in Spanish and 91.9% in English on RheumaMIR, below GPT-6 Sol at 99.4% (Holm-adjusted P<=0.005). GPT-6 Luna reached 97.7% and Clef 91.9-93.0%.
Accuracy was lower on vignettes: GPT-6 Sol 76.9%, GPT-6 Luna 63.5%, Clef 59.6%, Jev 50.0%, and Microsoft-Decision-1 40.4%, against a chance level of 9.6%. Each System One model scored below GPT-6 Sol.
On vignettes Jev became overconfident, with a mean probability of 0.659 and an over/underconfidence index of +0.159. Its answers at probability >=0.90 (25% of cases) were 84.6% correct, versus 95.5% for GPT-6 Luna at 42%.
Four System One models (Jev 1.13, Clef, Microsoft-Decision-1, Laya) and GPT-6 Luna via the OpenAI Decisions API were compared with GPT-6 Sol and two local generative models.
No detectable language differences were found. Jev scored 93.0% in Spanish and 91.9% in English on RheumaMIR, and the study reports no detectable language effects across the models tested.
High-confidence Jev predictions were less reliable on diagnostic vignettes than on examination questions, so the authors highlight the need for task-specific evaluation and calibration rather than assuming confidence transfers across tasks.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
An arXiv study comparing decision fine-tuning against reranker fine-tuning across six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, concluding that decision fine-tuning never produces the better reranker.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.