Article
CommunityJev Practices & Patterns
JevCoder: When Does Decision Delegation Pay Off in Code Repair? (arXiv 2610.19596)
A code-repair workflow that delegates bounded decisions to Jev as a low-cost probabilistic judge: across 400 SWE-bench tasks it cuts mean repair-model token use by 21.6-50.6% with higher observed repair rates on three models.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Evaluates three repair models on the same 400 tasks: 200 from SWE-bench Verified and 200 from SWE-Bench Pro Verified.
- JevCoder pairs code-snippet recommendations with action-type guidance and uses judgments to decide when to submit.
- Mean repair-model token use drops 21.6-50.6% against unguided repair, with higher observed repair rates on all three models.
- The study compares judges at each repair stage against no judge and against the repair model judging its own work.
- Its conclusion is about work, not judging cost: the benefit depends on what an intervention changes.
Watch out
arXiv preprint posted October 14, 2026 (cs.SE, cs.AI). Token counts cover the repair model, not the end-to-end cost including the judge, and no dollar figures appear in the abstract. Results cover three repair models on two SWE-bench splits, and the abstract names no code release.
Questions about this entry
Short answers drawn from the linked project or page.
JevCoder delegates bounded repair decisions to Jev as a probabilistic judge: code-snippet recommendations, action-type guidance for the next step, and the judgment that determines when to submit the repair.
Against unguided repair, JevCoder reduces mean repair-model token use by 21.6% to 50.6% across the three repair models tested, with higher observed repair rates in every case. The counts are the repair model's tokens, not the full pipeline.
Four hundred code-repair tasks: 200 from SWE-bench Verified and 200 from SWE-Bench Pro Verified. Each of the three repair models ran the same 400 tasks, so the comparisons share a workload.
The study compares delegated judgments with a workflow where the repair model itself acts as the judge, as well as with no judge at all. The abstract's finding is that a cheap separate judge changes repair work in ways self-judging does not reproduce.
That the payoff from delegating a decision depends on the work the intervention changes rather than on the judgment's own cost. A cheap judge alone is not the point; which decisions move the trajectory is.
The abstract names no repository or dataset release, so nothing beyond the paper is checkable from the listing. Treat reproduction as a from-scratch effort until the authors publish artifacts.
It is the coding-agent version of confidence-gated delegation: Jev judges bounded repair questions while the expensive model writes code, and policy in code decides what each judgment unlocks, such as submitting earlier or switching actions.
More Jev resources like this
A conceptual article that formulates a conditional Jevons hypothesis for machine evaluation: sufficiently large reductions in the marginal cost of usable machine evaluation may increase its organizational consumption where latent demand is substantial and complementary costs do not dominate.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Studies a fast learned actor in closed-loop Doom that hands control to a reasoning vision-language model only when a gate opens. On 900 held-out questions, deferring the least confident 30% of decisions gains +0.13 with doomLaya's option order, but no closed-loop variant reaches the exit.
Jev Practices & PatternsArticle#community#research#arxiv
Community
dexmac221/doomgemmaPython
dexmac221/doomgemmaPythonAn exploratory study of JevForest, a budgeted feature-acquisition policy that aggregates path-dependent proposals from bootstrapped trees and queries Jev for semantic answers. On small held-out samples it beats static and random orderings on AG News but not on TREC.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A report identifying an arithmetic-dependent rejection bottleneck in Jev: on paired arithmetic problems answer-present accuracy reaches 99% while correct rejection falls to 7%. A development-selected decision threshold lifts rejection from 7% to 79% while retaining 97% answer-present accuracy.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A behavioral diagnosis of why System One decision models choose wrong actions in agent loops. On a frozen benchmark of agent decision states, most wrong actions follow a correct reading of the state, and the more accurate the system, the more of its errors take this form.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A theoretical study of recalibrating many related yes/no questions that each have few labels, motivated by decision models such as Jev. It derives minimax compound-risk rates and shows pooled isotonic fitting with one per-question correction achieves them, while tuning questions separately loses a factor.
Jev Practices & PatternsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.