Article
CommunityJev Benchmarks & Evaluations
Confidence Is Not a Promise: A Reference Audit of Typed Decision Models
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Compares open-weight Laya across five checkpoints with Jev 1.13, which is reachable only through an API.
- Across 190 exchangeable evaluations, a point-estimate threshold breaks its promise in 147; Learn then Test keeps it in all.
- Calibrated in English, Jev holds a 95% promise in 8 of 51 locales.
- A fine-tuned Laya needs 250 to 8,000 labels to be non-inferior to Jev in four domains, and does not reach it on LIAR2.
- Confidence separates right from wrong answers at AUROC 0.80 to 0.96 but does not select faithful explanations.
Questions about this entry
Short answers drawn from the linked project or page.
It measures what a returned probability actually licenses. The authors argue a probability supports a statement about groups of decisions, not a single decision, and that a gate promising accuracy among the cases it decides alone keeps that promise only with a lower bound checked on data like its traffic.
No. The audit describes Jev 1.13 as available only through an API, while the comparison model, Laya, is open-weight and is evaluated in five checkpoints. The API also rounds reported values to two decimals, which the paper notes limits what an explanation can say.
Across 190 exchangeable evaluations, a threshold chosen by point estimate broke the promise in 147 of them. Learn then Test kept the promise in all of them, and a Beta lower-bound rule decided more items on average than Learn then Test in 128 evaluations and fewer in none, though without its guarantee.
Calibrated in English, Jev keeps a promise of 95% in 8 of 51 locales. The paper also reports that ordinary typing noise breaks a promise calibrated on clean text, and that in exploratory runs training with noise raised noisy accuracy with no loss on clean text.
In four further domains, a fine-tuned Laya needs from 250 to 8,000 labels to be non-inferior to Jev. On LIAR2 it does not reach non-inferiority at all.
No. Confidence separates right from wrong answers with an AUROC from 0.80 to 0.96, but the paper reports that it does not select faithful explanations. Exact coalition games of single decisions show what an explanation can still say through an API that rounds to two decimals.
The authors release the frozen answers with a datasheet, a promise check, and the code. The paper also reports drift results, where the preregistered claim for a Bayesian retraining loop was not supported, and M3 monthly series results where neither decider beat a constant answer on direction.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
An arXiv study comparing decision fine-tuning against reranker fine-tuning across six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, concluding that decision fine-tuning never produces the better reranker.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.