Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
CommunityJev Practices & Patterns

System Switch: When Should a Fast Decision Model Stop and Think? (arXiv 2610.09683)

Studies a fast learned actor in closed-loop Doom that hands control to a reasoning vision-language model only when a gate opens. On 900 held-out questions, deferring the least confident 30% of decisions gains +0.13 with doomLaya's option order, but no closed-loop variant reaches the exit.

See highlightsOpen arxiv.orgdexmac221/doomgemma
Stars
1
Language
Python
License
APACHE-2.0
Format
Article
Published by
Community
Last verified
Oct 2026
Tagscommunityresearcharxivagentscalibration
Works forJev for Decision gatingJev for Realtime controlJev for Evaluation
Used inJev for GamingJev for Robotics & HardwareJev for Software & DevTools

Highlights

  • On 900 held-out questions, 0.15B–9B decision models choose to collect items 1.6–1.8 times more often than chance among their errors.
  • Accuracy, calibration, and sensitivity are distinct: models of similar accuracy differ widely in AUROC.
  • Deferring the least confident 30% gains over random deferral in proportion to actor AUROC, with rank correlation 0.87.
  • With actor and rate chosen on held-out games the gain is +0.13 [0.08, 0.18], and +0.08 [0.02, 0.14] with shuffled options.
  • In closed loop over 33 games and three seeds, no variant reaches the exit.

Watch out

arXiv preprint submitted October 7, 2026 (cs.AI) by Gian Luca Bailo. Code, prompts, data, and logs are released; the closed-loop result is negative, no variant reaches the exit, and the reasoning model carries only about half of the offline deferral gain.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Code Owns the Simulation, Jev Owns the Evaluation (arXiv 2610.01834)
A test of Jev on reflection tests, one-shot matrix games, ALFWorld, and robot control that finds a sharp boundary: it succeeds when the right option can be judged from what the input describes, but fails when the right option depends on predicting something not in the input.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Can Jev be Your Q or Policy in Reinforcement Learning? (arXiv 2610.11692)
A study of whether Jev can serve inside reinforcement learning training rather than as a single-domain black box. Frozen Jev fills reference-policy, exploration-judge, and replay-rater roles, and training with it beats a standard RL learner across nine MiniGrid tasks and three Atari games.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops (arXiv 2610.18579)
A behavioral diagnosis of why System One decision models choose wrong actions in agent loops. On a frozen benchmark of agent decision states, most wrong actions follow a correct reading of the state, and the more accurate the system, the more of its errors take this form.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents (arXiv 2609.28940)
A position and case-study paper on using System One decision models like Jev and the open-source Laya as decision layers in LLM-driven penetration-testing harnesses. It names four decision points, runs an exploratory 13-vulnerability NeuroSploit comparison, and proposes Rave, an unevaluated model.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Dyad: Extending Large Language Models with Native Typed Decision-Making (arXiv 2609.36116)
Dyad augments a pretrained LLM with an environment-conditioned action encoder that scores candidate action descriptions to yield a distribution over typed actions. Training only the encoder helps across four unseen environments; joint training adds a 3.80% absolute gain on ALFWorld with a 9B model.
Jev Practices & PatternsArticle#community#research#arxiv
Community
When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev (arXiv 2609.39496)
A report identifying an arithmetic-dependent rejection bottleneck in Jev: on paired arithmetic problems answer-present accuracy reaches 99% while correct rejection falls to 7%. A development-selected decision threshold lifts rejection from 7% to 79% while retaining 97% answer-present accuracy.
Jev Practices & PatternsArticle#community#research#arxiv
Community

From the Jev guides

Original write-ups on what this entry is for.

What the First Month of Jev Papers Found
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
Measuring Jev: What Independent Benchmarks Actually Found
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
Designing for Rejection
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai