CommunityJev Practices & Patterns
Code Owns the Simulation, Jev Owns the Evaluation (arXiv 2610.01834)
A test of Jev on reflection tests, one-shot matrix games, ALFWorld, and robot control that finds a sharp boundary: it succeeds when the right option can be judged from what the input describes, but fails when the right option depends on predicting something not in the input.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Jev solves 99% of the counterintuitive Cognitive Reflection Test questions.
- In games it plays suboptimally as if its rational opponent acted at random, because the opponent's action is not given.
- In ALFWorld it favors commands that mention an object or place named in the task description.
- Asked separately what the opponent will do, Jev usually answers correctly; it fails when one call must simulate and evaluate.
- Given code-supplied lookahead or physics simulation, Jev becomes an expert controller through its evaluation ability.
Watch out
arXiv preprint posted October 1, 2026 and updated October 10, 2026 (cs.AI, cs.LG; 10 pages main text, 21 with appendix) by Yaodong Yang and co-authors. The boundary between evaluation and simulation is demonstrated on selected tests and environments, not established as a general rule for all agent tasks.
Questions about this entry
Short answers drawn from the linked project or page.
Jev succeeds when the right option can be judged from what the input describes, which the authors call evaluation, but fails when the right option depends on simulation, meaning predicting something not in the input.
It solves 99% of the counterintuitive Cognitive Reflection Test questions.
It plays as if its rational opponent acted at random, because the opponent's action is not given in the input.
Given the task to put a clean knife in the drawer, Jev carries an unwashed knife straight to the drawer instead of first washing it at the sink.
No. Asked separately what the opponent will do, Jev usually answers correctly and responds well given the opponent's action; it fails when one call must both simulate and evaluate.
Letting code make the prediction or simulation. When code supplies a lookahead in ALFWorld or physics simulation in robot control, Jev becomes an expert controller through its general evaluation ability.
Reflection tests, one-shot matrix games, the text game ALFWorld, and robot control.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Dan Makes Things drives an ESP32 robot with Jev decisions.
More Jev resources like this
Studies a fast learned actor in closed-loop Doom that hands control to a reasoning vision-language model only when a gate opens. On 900 held-out questions, deferring the least confident 30% of decisions gains +0.13 with doomLaya's option order, but no closed-loop variant reaches the exit.
Jev Practices & PatternsArticle#community#research#arxiv
Community
dexmac221/doomgemmaPython
dexmac221/doomgemmaPythonA study of whether Jev can serve inside reinforcement learning training rather than as a single-domain black box. Frozen Jev fills reference-policy, exploration-judge, and replay-rater roles, and training with it beats a standard RL learner across nine MiniGrid tasks and three Atari games.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Dyad augments a pretrained LLM with an environment-conditioned action encoder that scores candidate action descriptions to yield a distribution over typed actions. Training only the encoder helps across four unseen environments; joint training adds a 3.80% absolute gain on ALFWorld with a 9B model.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A behavioral diagnosis of why System One decision models choose wrong actions in agent loops. On a frozen benchmark of agent decision states, most wrong actions follow a correct reading of the state, and the more accurate the system, the more of its errors take this form.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A training-free method that studies redundancy in candidate evaluation for JEV-style multimodal decisions. Shared context anchoring, candidate prefix sharing, and decision-guided pruning cut candidate depth by 43.75%–45.83% while retaining 93.66%–97.52% of task scores.
Jev Practices & PatternsArticle#community#research#arxiv
Community
50GitHub stars
A StarCraft II controller combining fast JEV action selection with persistent GPT-6 planning that defeats the strongest non-cheating built-in AI, Lv7. The combined system wins four full games at Lv5-Lv7 at a mean estimated cost of USD 3.71 per game, with a median JEV response time of 0.422 s.
Jev Tools & CLIsArticle#community#research#arxiv
Community
sc2musa/Jev_StarPython
sc2musa/Jev_StarPythonFrom the Jev guides
Original write-ups on what this entry is for.
A game tick is a state plus a legal action list. How Clash Royale, Minecraft, and the earlier Mario, Pokémon, StarCraft, and drone loops ask Jev one step at a time.
Jev Demos & Experiments#gaming#agents#control
Read article
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
From the Jev community
Posts from builders shipping with Jev right now.