CommunityJev Practices & Patterns
Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops (arXiv 2610.18579)
A behavioral diagnosis of why System One decision models choose wrong actions in agent loops. On a frozen benchmark of agent decision states, most wrong actions follow a correct reading of the state, and the more accurate the system, the more of its errors take this form.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Pairs every next-action question with factual probes about the same state, each answered independently.
- Across seventeen fast systems and a reasoning model, most wrong actions follow correct reading.
- Error states recur across systems of every kind, so the failures are systematic rather than noise.
- Where probed facts determine the action, models that read every fact still fail to combine them.
- Composing the models' own answers in code repairs those failures; on real states the gap is often a missing sub-goal.
Watch out
arXiv preprint submitted October 13, 2026 (cs.AI) by Yubo Li, Ramayya Krishnan, and Rema Padman; the comment calls it an initial version sharing preliminary findings, with evaluations of more open-source System One models ongoing. Results come from a frozen behavioral benchmark, not from inspecting model internals.
Questions about this entry
Short answers drawn from the linked project or page.
Whether a wrong next action came from misreading the agent's state or from reading it correctly and still acting wrongly. The authors answer behaviorally rather than by inspecting internals, pairing each next-action question with factual probes about the same state.
A frozen benchmark of agent decision states drawn from simulated environments, customer-service policies, tool use, web navigation and coding. Each next-action question is paired with factual probes about the same state, answered independently rather than as part of the action choice.
Seventeen fast System One systems plus a reasoning model. The answer is consistent across them: most wrong actions follow correct reading, and the more accurate a system is, the larger the share of its errors that take this form.
No. The errors gather on the same states across systems of every kind, which the paper presents as evidence that they are systematic rather than noise. Where the probed facts provably determine the action, models that read every fact still fail to combine them.
In the cases where the probed facts provably determine the action, composing the model's own answers by code repairs the failure, showing the error sits in combining facts rather than in reading them.
Often not a fact but the next sub-goal: on real decision states the bottleneck is frequently deciding what to pursue next rather than resolving an unknown. The paper frames this as a distinct failure mode from misreading.
It is an arXiv preprint released October 13, 2026 as an initial version to encourage open discussion of preliminary findings, and the authors say evaluations of additional open-source System One models are ongoing, so the seventeen-system result should be read as preliminary.
More Jev resources like this
Studies a fast learned actor in closed-loop Doom that hands control to a reasoning vision-language model only when a gate opens. On 900 held-out questions, deferring the least confident 30% of decisions gains +0.13 with doomLaya's option order, but no closed-loop variant reaches the exit.
Jev Practices & PatternsArticle#community#research#arxiv
Community
dexmac221/doomgemmaPython
dexmac221/doomgemmaPythonA benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
HKUST-KnowComp/Jev-ToMPython
HKUST-KnowComp/Jev-ToMPythonA conceptual article that formulates a conditional Jevons hypothesis for machine evaluation: sufficiently large reductions in the marginal cost of usable machine evaluation may increase its organizational consumption where latent demand is substantial and complementary costs do not dominate.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A study of whether Jev can serve inside reinforcement learning training rather than as a single-domain black box. Frozen Jev fills reference-policy, exploration-judge, and replay-rater roles, and training with it beats a standard RL learner across nine MiniGrid tasks and three Atari games.
Jev Practices & PatternsArticle#community#research#arxiv
Community
107GitHub stars
A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.
Jev Practices & Patterns#community#typescript#agents
Community
vinilana/jev-eval-agentHTML
vinilana/jev-eval-agentHTMLA test of Jev on reflection tests, one-shot matrix games, ALFWorld, and robot control that finds a sharp boundary: it succeeds when the right option can be judged from what the input describes, but fails when the right option depends on predicting something not in the input.
Jev Practices & PatternsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
From the Jev community
Posts from builders shipping with Jev right now.