Article
CommunityJev Guides & Articles
LessWrong: testing Jev as a trusted monitor for AI control
A LessWrong pilot testing Jev as a trusted monitor in the ControlArena APPS backdoor setting: one yes/no question reaches AUROC 0.976 and catches 90% of backdoors at a 2% false-positive rate for about $0.04 per thousand submissions.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- Venkat T
- Last verified
- Oct 2026
Highlights
- A single yes/no Jev question scores AUROC 0.976 against LLM-written honest code and catches 90% of backdoors at a 2% false-positive rate.
- The author reports the result matches o3-mini with chain of thought (0.92-0.96 per the Control Tax paper) at about $0.04 per thousand submissions.
- Re-scoring the same code moved monitor scores by only 0.008 on average, and the top-10% audit set was 92% identical across calls.
- Adversarial comments, injected security approval and fake output tags did not lower suspicion on backdoored code by more than 0.014.
- A logistic regression on code size scored only 0.674 AUROC, which the author reads as evidence Jev is judging logic rather than length.
Watch out
Copy reviewed against the published LessWrong post (18 September 2026) on 2026-10-17. The study is a self-described pilot run on $5 of credit over 15 hours; the author ran no LLM baseline, so comparisons to published numbers are for positioning rather than measurement, and two of four initial hypotheses turned out wrong.
Questions about this entry
Short answers drawn from the linked project or page.
It tests TypeSafe's Jev as the trusted monitor of ControlArena's APPS backdoor task, where an untrusted model writes code that may hide a planted backdoor and a cheaper model scores how suspicious each submission looks.
AUROC 0.976 against the LLM-written honest set and 0.970 against the human-written set, with 95% confidence intervals of [0.968, 0.985] and [0.959, 0.979]; the two honest sets agree within the interval at every audit budget.
At a 2% false-positive rate, the main operating metric, it catches 9 out of 10 backdoors; true-positive rates at 1%, 2% and 5% FPR are 0.878, 0.895 and 0.928 on the LLM-written honest set.
The author says the 0.970 result sits inside and slightly above the 0.92 to 0.96 range the Control Tax paper reports for o3-mini with chain of thought, at about a hundredth of the price and with 0.34 s median latency.
Yes, per the post: re-scoring the same code moves Jev's scores by only 0.008 on average, and the top-10% audit set is 92% the same across all the calls, which is what makes a fixed threshold meaningful.
Suspicion on backdoored code was not lowered by more than 0.014 when comments instructed the monitor what to score, injected security approval or imitated output tags, consistent with Jev having no instruction channel to attack.
It is a pilot on $5 of credit over 15 hours with no baseline LLM run of its own, so published comparisons position the result rather than measure it; the author also notes two of four initial hypotheses turned out wrong in interesting ways.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Kubernetes admission control with a decision model, fail-closed.
Hacker News: SafeDep Gryph plus Jev checks for compromised coding-agent behavior.
2 pointsDetecting Compromised AI Coding Agents with Jev and GryphRead the thread on Hacker NewsX: Jev and Laya on security queries
X: Testing Jev for prompt injection
More Jev resources like this
HiddenLayer researchers tested Jev and an in-house System One-style model: on a simulated checkout page an attacker-controlled ad made the browser agent click away from the purchase, and TokenBreak-style edits lowered toxicity scores for prompts both models had flagged.
Jev Guides & ArticlesArticle#community#press#security
Community
HiddenLayer's insight post walks security teams through its Jev tests: on a simulated checkout page Jev gave the attacker ad's button 91% against 9% for Place order, and TokenBreak edits cut one prompt's toxicity score by more than two thirds so Jev stopped flagging it.
Jev Guides & ArticlesArticle#community#press#security
Community
Pillar Security analyzes Jev and System One models: a smart-home demo answered three typed questions in about 200 ms with no language model in the loop, while prompt scanning, output filtering and chain-of-thought monitoring all have nothing to inspect. TypeSafe's docs say state is not treated as hostile by default.
Jev Guides & ArticlesArticle#community#press#security
Community
Rogue Security's post uses Jev and System One models to frame agentic security as a decision problem: the same send_file action should be allowed, blocked or reviewed depending on who requested it and where the data goes, illustrated with policy case files rather than product results.
Jev Guides & ArticlesArticle#community#press#security
Community
Ian Maurer's ThinkThen analysis argues Jev's faster and cheaper are measurable today while its better is a bet on RLCD: unlike RLHF, which pleases humans, and RLVR, which optimizes benchmarks, Jev is trained to be reliable for programmatic use, in Diogo Almeida's own framing.
Jev Guides & ArticlesArticle#community#press#explainer
Community
A BIG CHANGE analysis of the Diogo Almeida Latent Space interview: Jev's most consequential proposition is the unit of automation, a bounded judgment inside an existing process, and the launch claim that Jev can't hallucinate needs a narrow reading because schema matching does not establish truth.
Jev Guides & ArticlesArticle#community#press#explainer
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.