Article
CommunityJev Benchmarks & Evaluations
The Value of Thinking: Learning When to Reason in Hybrid Decision Models
arXiv paper introducing hybrid decision models (HDM) with a self-routing block that estimates the value of thinking. Evaluates on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77, and MiniWoB; on JevBench HDM-4B scores 0.857 vs Jev's 0.851 at median 0.179s vs 0.652s.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Introduces zero-or-T thinking: a model either answers from implicit thinking in one forward pass or performs full autoregressive reasoning.
- Uses a hybrid decision model (HDM) with one frozen backbone and a lightweight self-routing block.
- Self-routing block estimates the value of thinking and runs slow mode only when value exceeds a per-request price λ.
- Introduces Calibrated Decision Policy Optimisation (CDPO) to train the block from episode-level returns when no per-decision label exists.
- On JevBench, HDM-4B scores 0.857 vs Jev's 0.851 (+0.7%) at median 0.179s vs 0.652s (-72.5%).
Questions about this entry
Short answers drawn from the linked project or page.
It explores zero-or-T thinking, where a single model either answers from implicit thinking in one forward pass, generating no token, or performs full autoregressive reasoning at a cost of T tokens when that is worth paying.
HDM is a setup where one frozen backbone serves both fast and slow modes, and a lightweight self-routing block chooses between them based on the estimated value of thinking.
It estimates the value of thinking, the utility the slow mode is expected to add over the fast one, from what the implicit thinking has already produced. The slow mode runs only where that value exceeds a price λ set per request.
CDPO is a reinforcement-learning recipe introduced for cases where no per-decision label exists, such as episodic agentic tasks. It trains the self-routing block from episode-level returns.
The paper evaluates HDM on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77, and MiniWoB.
On JevBench, HDM-4B scores 0.857 against Jev's 0.851 (+0.7%) at a median 0.179s against 0.652s (-72.5%).
On the agentic venue MiniWoB, HDM-27B cuts token cost by 80.1%.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Facebook: Intake Router keeps diagnosis out of the answer space at the front desk.
FacebookClinicnotes
X: OpenRouter Decision Model Rankings
X: herdr + jev, an agent router
X: A group chat bot with decision room routing
More Jev resources like this
arXiv paper presenting RouteAlign, a method that uses task requirements to structure model performance measurement and LLM service selection. Reports quality gains over open Jev-style decision backends at lower generation cost.
Jev Benchmarks & EvaluationsArticle#arxiv#routing#evaluation
Community
arXiv paper comparing three decision models and three general LLMs as actors and judges on ALFWorld, and using JEV as a teacher on ALFWorld and WebShop. Reports GRPO policy gains and releases companion code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#agents
Community
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper testing whether Jev, a decision model that outputs judgment probabilities, works as a multilingual judge. It compares Jev with generative LLM judges across four annotated benchmarks, three evaluation formats, and up to 47 language varieties.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper that re-implements five readouts for typed-decision models and trains them under one protocol: the same backbone, 15 public English training sources, a per-head learning-rate sweep, and the same seed. It reports accuracy, inference, and training comparisons across Qwen3 backbones.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
SO-Arena is a benchmark for System One models covering 11 domains and five capability families. It measures capability spectrum and decision reliability across 21 models, including decision stability under option-set changes and missing-answer recognition.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.