Article
CommunityJev Benchmarks & Evaluations
MedJev: Evaluating Zero-Shot System-One Decision Making in Medicine
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- MedJev is an open System One model that runs locally on a laptop.
- Evaluated on ten medical benchmarks covering knowledge, diagnosis, and patient-level prediction with MIMIC IV and MIMIC IV ED.
- Scores 72.5% on MedQA, 75.4% on USMLE, and 80.1% on NEJM with ECE between 0.028 and 0.063.
- In-hospital mortality AUROC of 0.926 exceeds XGBoost at 0.902, but ED triage admission trails supervised models, 0.745 vs 0.809.
- A two-parameter recalibration using 50 to 200 labeled examples reduces ECE to 0.007.
Questions about this entry
Short answers drawn from the linked project or page.
MedJev is an open System One decision model that answers structured questions in a single forward pass without generating text. The paper describes it as running locally on a laptop, which the authors present as a low-computation option for clinical decision support.
Ten medical benchmarks spanning medical knowledge, diagnosis, and patient-level prediction. Patient-level tasks use MIMIC IV and MIMIC IV ED data, and results are compared against zero-shot LLM baselines, published GPT results, and supervised models.
MedJev reaches 72.5% on MedQA, 75.4% on USMLE, and 80.1% on NEJM, which the authors describe as approaching published GPT-4 performance while keeping well-calibrated confidence estimates with ECE between 0.028 and 0.063.
Knowledge and diagnosis tasks are well calibrated, with ECE from 0.028 to 0.063. Patient-level predictions are substantially miscalibrated, with in-hospital mortality showing an ECE of 0.286 before any recalibration is applied.
On operational outcomes. For ED triage admission MedJev scores 0.745 against 0.809 for supervised models. It does exceed XGBoost on in-hospital mortality, with an AUROC of 0.926 versus 0.902.
The paper introduces a lightweight, two-parameter recalibration method that adapts predictions to local patient populations using 50 to 200 labeled examples. Applied to patient-level predictions, it reduces ECE to 0.007.
Yes. The authors state that they release their code and evaluation harness to support future research, alongside the reported benchmark results for MedJev and the comparison baselines.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
An arXiv study comparing decision fine-tuning against reranker fine-tuning across six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, concluding that decision fine-tuning never produces the better reranker.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.