CommunityJev Practices & Patterns
Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents (arXiv 2609.28940)
A position and case-study paper on using System One decision models like Jev and the open-source Laya as decision layers in LLM-driven penetration-testing harnesses. It names four decision points, runs an exploratory 13-vulnerability NeuroSploit comparison, and proposes Rave, an unevaluated model.
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Proposes four decision points: finding adjudication, severity recalibration, agent pruning, and confirmation loops.
- The NeuroSploit case study compares one run with TypeSafe System One (Jev) and one without, on a web target with 13 vulnerabilities.
- Reviews published specifications for Jev, Jev-Ultrafast, and open-source Laya, without assuming benchmark results transfer to pentesting.
- Discusses RLHF, RLAIF, RLCD, and RLHV training approaches and their implications for trust in security decisions.
- Rave, the proposed domain-adapted model, is outlined with training data and an evaluation protocol but not evaluated.
Watch out
arXiv preprint posted September 24, 2026 (cs.CR, cs.AI, cs.SE), 21 pages, single author. The NeuroSploit comparison is exploratory with one run per condition, and the author states the differences do not establish statistical significance; Rave is a proposal without results, and the model review relies on published specifications.
Questions about this entry
Short answers drawn from the linked project or page.
LLM harnesses often use the same models for reconnaissance, exploitation, reporting, and confirmation, which can cause false positives, inflated severity, and wasted compute; typed calibrated verdicts are proposed as a separate layer.
Finding adjudication, severity recalibration, agent pruning, and confirmation loops, each of which the paper argues could be delegated to a non-generative decision model.
It compared one run with TypeSafe System One (Jev) and one run without it, against a web target containing 13 vulnerabilities, examining severity distribution, runtime, and grading by exposed data type.
No. The author says the differences motivate the architecture but do not establish statistical significance, because it is an exploratory comparison with a single run per condition.
It reviews published specifications for Jev, Jev-Ultrafast, and the open-source Laya, explicitly without assuming that results from other benchmarks transfer to penetration testing.
Rave is a proposed domain-adapted System One model for pentest harnesses, with its training data, evaluation protocol, and potential effect on harness assurance outlined, but no reported results.
The paper discusses RLHF, RLAIF, RLCD, and RLHV as training approaches and what each implies for trusting decisions made by calibrated models in security workflows.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsHacker News: Show HN: Agent Chaperone screens tool calls and results with Jev before the agent sees them.
sepehrsafari5 pointsShow HN: Agent Chaperone – Screen AI agent tool calls and results with JevRead the thread on Hacker NewsX: Secret detection with Jev, tested
Related terms
Glossary definitions related to this entry.
More Jev resources like this
0GitHub stars
A systematic study of Jev's security and privacy implications using the official API and NanoJev, a local model with controllable training data. It adapts prompt injection, adversarial suffixes, and poisoned training data, runs inference attacks against typed outputs, and maps dual uses.
Jev Practices & PatternsArticle#community#research#arxiv
Community
shihe98/Security_Privacy_JevPython
shihe98/Security_Privacy_JevPythonA pre-registered evaluation of Jev as a one-second, training-free screen for SCADA alarm triage in water distribution networks. On a four-class EPANET C-Town benchmark it matched a hand-written rule tree, and a gated cascade spared an LLM reviewer 35-38% of windows.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Studies a fast learned actor in closed-loop Doom that hands control to a reasoning vision-language model only when a gate opens. On 900 held-out questions, deferring the least confident 30% of decisions gains +0.13 with doomLaya's option order, but no closed-loop variant reaches the exit.
Jev Practices & PatternsArticle#community#research#arxiv
Community
dexmac221/doomgemmaPython
dexmac221/doomgemmaPythonJevOut shows short background context additions can flip correct decisions in four systems including Jev across seven datasets. Within 64 accepted evaluations per decision, additions redirect 61.4%-73.2% of initially correct decisions toward a wrong option fixed in advance.
Jev Practices & PatternsArticle#community#research#arxiv
Community
xzx34/JevOutPython
xzx34/JevOutPythonA report identifying an arithmetic-dependent rejection bottleneck in Jev: on paired arithmetic problems answer-present accuracy reaches 99% while correct rejection falls to 7%. A development-selected decision threshold lifts rejection from 7% to 79% while retaining 97% answer-present accuracy.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A system that codes Texas crash narratives as gated, typed Jev decisions returning probabilities and no text: a screen covered 499,500 narratives and 195,857 were coded with a 27-question schema. The typed model attains F1 of 0.908 against human labels, and recalibration cuts calibration error 3.3-fold.
Jev Practices & PatternsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
From the Jev community
Posts from builders shipping with Jev right now.