Article
CommunityJev Benchmarks & Evaluations
Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents
arXiv paper comparing three decision models and three general LLMs as actors and judges on ALFWorld, and using JEV as a teacher on ALFWorld and WebShop. Reports GRPO policy gains and releases companion code.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Compares three decision models and three general LLMs as actors and judges on ALFWorld.
- All six models as actors are pulled away from a subgoal that must come first by an easier one.
- As judges seeing the whole trajectory, the same models credit steps toward that subgoal.
- JEV as teacher raises a Qwen2.5-1.5B GRPO policy from 72.8% to 82.9% on ALFWorld.
- On WebShop the same teacher setup lifts success from 56.8% to 71.4%.
Questions about this entry
Short answers drawn from the linked project or page.
It asks whether a decision model that acts poorly can still teach well, noting that prior work has studied decision models mainly as actors and judged their usefulness by how well they act.
Actors and judges are compared on ALFWorld, and JEV is used as a teacher on both ALFWorld and WebShop.
It raises success from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop.
Asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions when the judge sees the whole trajectory.
All six models, three decision models and three general LLMs, were pulled away from a subgoal that must come first by an easier one.
The paper states that code is available in the ZJU-REAL/decision-agents repository on GitHub.
Judging an action after the episode ends reveals the consequences that acting must anticipate, so a model that acts poorly can still score actions usefully for reinforcement learning.
More Jev resources like this
arXiv paper introducing S²-MAS, a society-scale multi-agent simulation framework built on 500 million synthetic household personas from survey data. Its Genese engine separates decisions from state updates, and sharp-421m-mlx is benchmarked against Jev 1.13 on financial-choice tasks.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper that re-implements five readouts for typed-decision models and trains them under one protocol: the same backbone, 15 public English training sources, a per-head learning-rate sweep, and the same seed. It reports accuracy, inference, and training comparisons across Qwen3 backbones.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
SO-Arena is a benchmark for System One models covering 11 domains and five capability families. It measures capability spectrum and decision reliability across 21 models, including decision stability under option-set changes and missing-answer recognition.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
arXiv paper introducing hybrid decision models (HDM) with a self-routing block that estimates the value of thinking. Evaluates on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77, and MiniWoB; on JevBench HDM-4B scores 0.857 vs Jev's 0.851 at median 0.179s vs 0.652s.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper presenting AbductionBench, a unified framework and taxonomy for abductive reasoning tasks, instantiated on 27 datasets across ten domains and used to evaluate eight LLMs from four model families on single-choice explanation selection.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.