Article
CommunityJev Benchmarks & Evaluations
Cross-Primitive Consistency in Jev
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Introduces PrimaCons, a four-axis benchmark for cross-primitive consistency that preserves query semantics.
- Evaluations across three Jev sizes reveal consistency violations on all four axes.
- Cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91.
- CPRC attaches a lightweight residual network to a frozen backbone, jointly optimizing accuracy and consistency.
- Under leave-one-domain-out, CPRC reduces cross-primitive inconsistency by 58.57% and improves accuracy by +2.97%.
Questions about this entry
Short answers drawn from the linked project or page.
PrimaCons is a four-axis benchmark introduced in the paper for assessing cross-primitive consistency in Jev. It evaluates whether the same semantic state, expressed through different typed querying primitives such as binary and multiple-choice queries, yields consistent decisions while preserving the semantics of the query.
The paper reports evaluations across three Jev sizes. Consistency violations were observed across all four axes of the PrimaCons benchmark for those evaluated models.
The analysis shows that cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91, and that the observed consistency violations follow systematic patterns rather than appearing randomly.
CPRC is the mitigation proposed in the paper. It attaches a lightweight residual network to a frozen backbone and jointly optimizes predictive accuracy and cross-primitive consistency, rather than retraining the underlying model.
Under a leave-one-domain-out protocol, CPRC reduces cross-primitive inconsistency by 58.57% and improves accuracy by +2.97%. The paper reports that these gains hold in both in-domain and out-of-domain settings.
The study uses a leave-one-domain-out protocol to test CPRC, reporting results for both in-domain and out-of-domain settings alongside the four-axis PrimaCons consistency measurements.
The paper states that code and the benchmark are available in the public GitHub repository 67L1/PrimaCons-CRPR, referenced from the arXiv abstract page for this work.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
An arXiv study comparing decision fine-tuning against reranker fine-tuning across six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, concluding that decision fine-tuning never produces the better reranker.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.