Article
CommunityJev Benchmarks & Evaluations
Benchmarking System One Models for Time-Series Decision Making: A Case Study of Jev
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
- Format
- Article
- Reading time
- 1 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Uses Jev 1.13.0 as the case study model for structured System 1 decision making on time-series tasks.
- Covers time-series classification, event type prediction, oil temperature direction, and stock movement prediction.
- Continuous forecasting targets are converted into discrete decisions to fit the model interface.
- Reports stronger performance on selected explicit temporal patterns and on event type prediction.
- Finds near-chance discrimination on many numerical sequence classification tasks and stock movement prediction.
Questions about this entry
Short answers drawn from the linked project or page.
It evaluates Jev 1.13.0, treated as a structured System 1 model that directly outputs answers and their probabilities, on time-series decision tasks, using a unified framework scored with accuracy and AUC.
Four settings: time-series classification, event type prediction, oil temperature direction, and stock movement prediction.
A unified evaluation framework reports accuracy and AUC, and some datasets also reveal a gap between ranking quality and final decision accuracy.
Continuous forecasting targets are converted into discrete decisions so they fit the model interface, which outputs an answer together with its probability.
Results show stronger performance on selected explicit temporal patterns and on event type prediction.
It shows near-chance discrimination on many numerical sequence classification tasks and on stock movement prediction, which the authors attribute to open issues in temporal representation, domain adaptation, and decision-score alignment.
Yes. Standardized sample construction, model calls, and result recording are set up so the evaluation can be extended to future models.
More Jev resources like this
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Evaluation of MedJev, an open System One model that runs locally on a laptop, across ten medical benchmarks for knowledge, diagnosis, and patient-level prediction on MIMIC IV and MIMIC IV ED, with a two-parameter recalibration method and a released evaluation harness.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
An arXiv study comparing decision fine-tuning against reranker fine-tuning across six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, concluding that decision fine-tuning never produces the better reranker.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.