Jev on 100 real agent tool calls

That is the bar the post sets, and it matters because 79% of these tool calls are harmless local commands, so a constant benign label already scores 79%. Jev reaches 93% zero-shot and 95% with nine shots. The caveat cuts both ways: some disagreements were dropped because the labeling rules were themselves unclear.

Jev, on this set: 7 of 9 dangerous calls flagged against 4 of 9 for Sonnet 5, even though Sonnet 5 scored higher overall at 98% versus 93% zero-shot. On a security boundary, where a miss lets data leave the perimeter, the miss count matters more than the headline accuracy.

Too thin to quote as a rate. The post says Jev made no errors at confidence of at least 0.7, but that comes from nine dangerous calls inside 100 samples, called early results with the dataset and prompts still being refined. Treat it as a threshold to test on your own traces.

Partly. The post's hardest case separates what a tool returns from what must be true before it may run: a Slack send returns a trivial ack carrying no secrets, yet delivers internal bytes to a channel. Models routinely confuse the two, and the authors concede part of that is prompt design on their side rather than model failure.

Four labels for an Information Flow Control check: delta_audience, who can see the tool output as self, internal or public; delta_trust, whether that output is trusted or suspicious; requires_audience, who can receive the data the call sends; and requires_trusted, whether to block when the session already holds untrusted data. Dynamic tools such as bash and custom MCP calls cannot have contracts defined in advance, so this runs per call.

A zero-shot Qwen3-Reranker-4B baseline that scores each tool-call and label pair separately and softmaxes across results, needing one forward pass per label but never seeing the options together, so it avoids option-order bias. Also Laya on ModernBERT-large, SemIf (formerly OpenJev) on Qwen and MiniCPM backbones, Bespoke-Nimble-9B as a Qwen3.5-9B fine-tune, and Sonnet 5.

No free-form prompt. You pass structured state such as the tool name and its arguments, plus a list of typed questions each carrying an id, a type and criteria; the questions are answered in parallel. The motivation is stated plainly in the post: routing should not cost two seconds and a pile of tokens to produce a label.