Article
CommunityJev Benchmarks & Evaluations
Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-Makers
arXiv study evaluating Jev as a decision reader of truncated LLM reasoning across seven reasoners and five multiple-choice benchmarks, comparing accuracy and estimated cost against forced answering at budgets from 500 to 16,000 tokens.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Covers seven LLM reasoners and five multiple-choice benchmarks, giving 35 reasoner-benchmark pairs.
- Jev beats forced answering at tested budgets from 500 to 16,000 tokens, including estimated reasoning and answering costs.
- At a 1,500-token budget it improves accuracy by 4.1 pp overall and 8.6 pp on truncated questions.
- Offline replay estimates 34.5% of the full-run cost with a 1.13 pp accuracy loss.
- A policy selected on training folds saves 71.3% of estimated full-run cost on held-out folds, losing 0.58 pp.
Questions about this entry
Short answers drawn from the linked project or page.
It is a model that selects among candidate options without producing a reasoning response. The paper evaluates Jev in that role, reading truncated reasoning traces instead of forcing an answer from an incomplete trace.
At a 1,500-token budget Jev improves accuracy by 4.1 pp overall and 8.6 pp on truncated questions, with significant gains in 17 pairs under Holm-corrected bootstrap tests.
Seven LLM reasoners and five multiple-choice benchmarks, for 35 reasoner-benchmark pairs, at tested budgets from 500 to 16,000 tokens.
Offline replay estimates 34.5% of the full-run cost with a 1.13 pp accuracy loss. A policy selected on training folds saves 71.3% of the estimated full-run cost on held-out folds, with a 0.58 pp accuracy loss.
Jev shows strong question-only accuracy, exceeding 500-token forced answering on average at 68.0% versus 65.5%, although the advantage varies by benchmark.
Gains are reported at tested budgets from 500 to 16,000 tokens, and the question-only advantage varies by benchmark. Cost figures come from offline replay and from a policy selected on training folds rather than a full production deployment.
No. System-1 decision models select among candidate options without producing a reasoning response, which is why the paper frames Jev as a complement to LLM reasoning rather than a replacement for it.
More Jev resources like this
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper measuring gender bias in decision models, testing TypeSafe's closed Jev and five open models on gender-swapped yes/no decisions, BBQ, StereoSet, and Fulgu and Capraro's gender-bias studies, with results on top-answer flips and confidence-gate effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Reference audit of typed decision models that measures what a returned probability licenses, comparing open-weight Laya checkpoints with Jev 1.13 over 190 exchangeable evaluations, locale coverage, drift, and explanation limits. Releases frozen answers, a datasheet, a promise check, and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.