Article
CommunityJev Benchmarks & Evaluations
Can JEV Support Clinical Trial Criteria Assessment?
Retrospective study comparing JEV, a non-generative decision model returning native option probabilities, with schema-constrained Qwen Flash on 2,048 patient-condition pairs from TrialGPT and MIMIC-III/IV under Zero Data Retention policies.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Compared JEV with schema-constrained Qwen Flash on 2,048 test patient-condition pairs.
- Batch-mode macro-F1: TrialGPT 0.652 vs 0.660; MIMIC-IV 0.807 vs 0.814; MIMIC-III 0.768 vs 0.534.
- Recall for conditions labeled as having insufficient information was 56.3% for JEV vs 9.9% for Qwen.
- Median job latency was 1.18-1.40 s for JEV vs 2.80-8.90 s for Qwen.
- Estimated cost per condition was 53%-74% lower; all evaluation ran under Zero Data Retention policies.
Questions about this entry
Short answers drawn from the linked project or page.
It retrospectively compared JEV, described as a non-generative decision model returning native option probabilities, against schema-constrained Qwen Flash on clinical trial prescreening, using patient records, criterion questions, and predefined answer options.
2,048 test patient-condition pairs drawn from TrialGPT and from question-answer datasets derived from MIMIC-III and MIMIC-IV, with performance measured against reference labels.
Macro-F1 was similar on TrialGPT (0.652 vs 0.660) and MIMIC-IV (0.807 vs 0.814), while JEV substantially outperformed Qwen on MIMIC-III (0.768 vs 0.534, delta +0.234, 95% CI 0.187-0.281).
The authors attribute it largely to higher recall for conditions labeled as having insufficient information: 56.3% for JEV versus 9.9% for Qwen.
Across all three datasets JEV had lower observed median job latency (1.18-1.40 s vs 2.80-8.90 s) and 53%-74% lower estimated cost per condition.
The comparison was evaluated under strict Zero Data Retention (ZDR) policies intended to protect clinical data privacy.
JEV is positioned as an initial assessment component giving criterion-level judgments and flagging potential information gaps; observed errors in interpreting clinical evidence mean outputs should be verified against patient records before informing eligibility decisions.
More Jev resources like this
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper introducing GDGraph, a guideline-derived decision graph framework that compiles radiology guidelines into an executable graph and has a System One model answer localized clinical questions, reporting 90.8% accuracy across 347 cases.
Jev Benchmarks & EvaluationsArticle#arxiv#research#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.