Article
CommunityJev Benchmarks & Evaluations
Thinking Fast, Trading Late: System One Decision Models vs the Market
arXiv paper proposing OLIVIA, a protocol for look-ahead-free tests of AI decision models, applied to probability-returning models reading S&P 500 news with masked company identities, validation-only selection, and a hash-frozen analysis plan.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- OLIVIA uses point-in-time inputs with company identities masked, selection on validation data only, and an analysis plan frozen by cryptographic hash.
- The fine-tuned open-weights model scored daily information coefficients of 0.055 and 0.065 on two validation quarters.
- On the blinded January-July 2026 test that signal fell to 0.020 (t = 1.7).
- Commercial models' readings line up with the overnight move to the open (information coefficient 0.074 to 0.077) but not with the move after it.
- Commercial models identified masked companies in 62% to 74% of cases against a 28% baseline.
Questions about this entry
Short answers drawn from the linked project or page.
OLIVIA is a protocol for testing whether news-reading predictability survives look-ahead, memorization and selection, using point-in-time inputs with company identities masked, selection on validation data only, an analysis plan frozen by cryptographic hash, and a test period unblinded once through a logged lock.
An open-weights model fine-tuned on 2024-2025 data, three commercial models, and standard baselines, all reading news about S&P 500 stocks.
The paper defines them as non-generative language models that return probabilities for typed questions, as opposed to models that generate text.
It replicated on two validation quarters with daily information coefficients of 0.055 and 0.065, but fell to 0.020 (t = 1.7) on the blinded January-July 2026 test.
No. Commercial models identified masked companies in 62% to 74% of cases against a 28% baseline, so masking hides names rather than companies.
No. The paper reports that no pre-registered strategy survives costs and a deflated Sharpe ratio, and that no model can tell over- from under-reaction.
Commercial models' readings align with the overnight move to the open (information coefficient 0.074 to 0.077) but not with the move after it, so the news is already priced by the time a trader can act at the open.
More Jev resources like this
arXiv paper presenting the first systematic bias and fairness audit of Jev, a typed-decision model for recruitment. It measures how demographic attributes and candidate cues shift screening scores, rankings, and shortlisting, against a noise floor, plus validity, latency, and cost.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#research
Community
arXiv evaluation of 14 System-1 decision models, including Jev, against four generative judges on eleven judge benchmarks, covering accuracy, calibration, cost, and robustness to instructions injected into judged responses.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study evaluating Jev as a decision reader of truncated LLM reasoning across seven reasoners and five multiple-choice benchmarks, comparing accuracy and estimated cost against forced answering at budgets from 500 to 16,000 tokens.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.