Article
CommunityJev Benchmarks & Evaluations
Structured Clinical Memory as Context for Probabilistic Decision Models
arXiv paper testing whether compact structured memory improves a probabilistic decision model's answers on 30 fully synthetic clinical cases across twelve domains, evaluated with 120 typed questions and reporting accuracy, log loss, Brier score, token and latency costs.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- 30 fully synthetic cases spanning twelve clinical domains, each paired with synthetically generated structured memory.
- 120 typed questions evaluated with Jev; raw-vignette accuracy 90.0% versus 93.3% with memory added.
- Choice log loss fell from 0.258 to 0.175 and binary Brier score from 0.054 to 0.022.
- Memory added about 310 input tokens per call on average; observed mean latency stayed near 0.5 seconds.
- Five questions flipped from incorrect to correct and one the other way (exact McNemar p=0.219).
Questions about this entry
Short answers drawn from the linked project or page.
It asks whether a compact structured memory can improve a probabilistic decision model's answers when the underlying patient vignette is already available, using 120 typed questions evaluated with Jev.
The authors created 30 fully synthetic cases spanning twelve clinical domains and evaluated 120 typed questions with Jev, comparing a raw-vignette arm against an arm with structured memory added.
Accuracy rose from 90.0% (95% CI 84.6%-95.4%) on the raw vignette to 93.3% (88.9%-97.8%) with memory. Five questions changed from incorrect to correct and one changed in the opposite direction.
Memory added 310 input tokens per call on average, while observed mean latency was unchanged at approximately 0.5 seconds.
No. The authors describe the results as directionally positive rather than clinically validating: the cases are synthetic, the labels are single-author judgments, and the memory is an oracle upper bound rather than extracted production output.
Each synthetic case was paired with synthetically generated structured memory, which the paper treats as an oracle upper bound rather than output extracted from a production pipeline.
It supports evaluating structured clinical memory as a reusable state interface for decision models, rather than treating the pilot as evidence of clinical performance.
More Jev resources like this
arXiv paper testing whether Jev can rank candidate demonstrations for in-context learning by judging their utility for a given query, and distilling that judgment into a lightweight MLP to cut the cost of large-scale retrieval.
Jev Benchmarks & EvaluationsArticle#arxiv#research#evaluation
Community
arXiv paper introducing GDGraph, a guideline-derived decision graph framework that compiles radiology guidelines into an executable graph and has a System One model answer localized clinical questions, reporting 90.8% accuracy across 347 cases.
Jev Benchmarks & EvaluationsArticle#arxiv#research#evaluation
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.