AI control assumes a stronger, less trusted model does the work while a weaker, trusted model watches. The monitor need not be smarter than the actor; it need only be good at a narrow question, cheap enough to run often, and honest about uncertainty. A typed decision model fits that shape well, provided you do not mistake a monitor for a sandbox.
What a trusted monitor must do
A monitor observes a trace or a proposed action and returns a judgment that a policy can use. Three properties matter more than raw accuracy.
First, it must be cheap enough for the required frequency: a retrospective trace judge can take a second per trajectory; an inline gate cannot. Second, it must return a calibrated confidence, because the point is to route uncertain cases to a more expensive check. Third, it must fail in a known direction; a monitor that fails open is not a control.
Jev's output shape helps with the first two: probabilities per option, no generated text, and a confidence statistic derived from the distribution. The third is a design choice, not a model property.
Trace review evidence
JEV as a Judge for Agent Trace Security is the most direct evaluation of Jev as a retrospective monitor. It compares Jev with four generative judges across four benchmark collections totaling 5,219 agent trajectories, all under a common risk rubric.
Jev reaches a benchmark-averaged positive-class F1 of 77.8, against 74.1 for the strongest generative configuration, GLM-5.2. Valid-result coverage is 95.5% for Jev and 94.4% for the generative judges. Median successful-call latency is 0.99 seconds, with an estimated token cost of $0.000195 per valid judgment. Jev leads on ATBench500 and MCPHunt; the GLM judges lead on R-Judge and TraceSafe.
The caveats matter. The comparison is self-run on four public collections under one rubric, so it may not transfer to other trace formats or policies. The authors position Jev as an economical screening signal with trade-offs in precision and recall, not as a replacement for a generative judge. That is the right role: use the cheap monitor to clear most traffic and escalate the rest.
Zero-shot failure detection
Just Ask Jev tests a broader claim: can a single generic question detect alignment failures it was not trained on? It covers ten failures, from sycophancy and jailbreaks to reward hacking and power seeking, across 44 benchmarks and five target models.
A single generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks. Question wording mattered little; context mattered more, mostly through fields that encode the label. Jev matches the reference scorer's agreement with human labels and costs 63 times less than LLM-judge scorers. The paper also surfaces label defects in existing benchmarks.
The caveats: the authors designed the benchmark and evaluate Jev on it; ground truth comes mostly from benchmark scorers, with humans used on two; and AUROC depends on existing labels. It is strong evidence that a typed judge can detect failures zero-shot, not evidence that it will detect your failures.
Accept or escalate
JEV-as-a-Judge turns the monitor into a cascade. Jev returns label probabilities; a pre-frozen confidence threshold decides whether to accept the verdict or send the case to a reasoning judge. Against sixteen generative and reward-model judges with blinded human adjudication, it comes within three points of GPT-6 wherever the verdict can be read off the text, at 0.36% of GPT-6's fee and a median latency of 0.15 seconds.
It falls behind where the verdict must be derived, as in math, code, and logic. Confidence marks where the cheap judge should stop, which makes escalation viable rather than a blanket replacement.
The cascade result is the practical one: with a threshold frozen in advance, accepting confident verdicts and escalating the rest was 0.9 points more accurate than GPT-6 on 1,610 held-out pairs, at 41% of its fee. In a pre-specified live test on two new workloads, it matched GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose, and the authors recommend validating thresholds locally instead of trusting a global setting.
The memory-gate counterexample
Suggested Is Not Superseded is the cautionary case. It evaluates Jev as a memory-retirement gate: when should a new utterance retire an old stored claim? The test set is 500 pairs built from STALE, labeled by three LLM annotators and two human adjudicators, one an author.
Of 179 pairs the benchmark marks as changed, only 72 meet the contract's bar for an established change. The binary gate never retired a claim across 141 same-conversation non-changes, which sounds excellent until you see the other side: it prematurely retired 39.2% of claims whose change was only suggested. A three-way contract (superseded, needs confirmation, compatible) cut that to 15.5%, but imposed a 15.3-point detection cost, missing the pre-registered 10-point budget. A development-selected escalation cascade was indistinguishable from simply raising the threshold.
The lesson is not that Jev is bad at memory. It is that a binary score wired to an irreversible action is the wrong design. Route suggested changes to pending evidence instead of deletion. The memory-gate glossary entry describes the pattern.
Inline monitors
Jev Sentinel puts the monitor in three places in a coding-agent session: before a tool call, where Jev scores intent (on task, off task, injected, needs context) and risk from 0 to 2; before the agent reads a tool output, which is scanned for instructions aimed at AI agents; and after a reply, for harmful content or relayed instructions. A flag adds a warning and requires approval for later actions until a reset. Secrets are scrubbed before anything reaches Jev, and the guard fails closed: if Jev errors or no key is set, it asks.
jev-guard asks three typed questions before every tool call: harm, whether the user requested it, and whether it came from untrusted content. Destructive calls are denied. After a tool returns, it scans for prompt injection and canaries and remembers hits for the session. A typical call is about 1k tokens, roughly $0.00004 per tool call, with measured p50 around 580 ms through the Vercel AI Gateway. On a machine with 662 installed skills, none crossed the flag line, while planted exfiltration samples scored 0.99. It is a guardrail, not a sandbox, and host support is uneven.
agi-jev-containment shows the incident side. It classifies chains of tool events rather than single calls, with Sentinel rules and SafetyDrift, and sets escalate-only levels. L1-L2 record, L3 tags and continues, L4 contains one agent and pages on-call, and L5 environment cut waits for a human. Host scripts exist but never auto-run. The README calls it a laptop-scale demo, not production IAM.
Thresholds, logs, and the limits of trust
A monitor's confidence is not a correctness guarantee. TypeSafe's confidence documentation is explicit that confidence is a 0-1 statistic derived from the distribution, that Noul answers carry none, and that thresholds depend on the domain and the model's performance on your workload. Start conservative and tune against your own data.
jev-consensus-gate is a useful pattern for the irreversible end: ask safety, legitimacy, and adversarial framing independently, require all three to allow, let any single block veto, and route any escalate to a human. It costs about $0.00006 per action.
systemone-probe is the conformance check: contract, correctness, calibration, determinism, batching, robustness, and limits. It will not tell you whether the monitor catches your incident class, but it will tell you whether the endpoint honors the answer contract.
The decision-gates article covers the policy side, the guardrails article covers the input boundary, and the measuring article covers evaluation design. A trusted monitor is a useful component. It is not a substitute for the system that decides what happens when the monitor says no.
