We tested Jev on 100 real agent tool calls

The constant-benign baseline is 79%, so beating do-nothing is about 14 points, not 93. That framing comes from the vendor's own write-up, and it is the honest number to quote when someone asks how good the results are.

Barely. The fetched page is the title plus site chrome with no comments, and the entry notes the discussion adds little beyond the article's headline figures. Sample construction, label definitions and the per-model table live in the article.

The entry describes production coding-agent tool calls, so the inputs are real rather than invented, which is the write-up's strongest point. The matching weakness is that a benign-heavy corpus makes a high score cheap to earn.

Not documented here. A gate that fires on benign tool use has a real cost in blocked work, and neither the thread nor the entry copy lists precision or a false-positive rate. You need the article's full table to judge whether it is safe to automate.

Posted by arseny_info with 12 points on 21 September 2026, linking the Archestra article archestra-jev-tool-calls. The fetched page is the title plus site chrome with no comments attached, so the thread contributed no discussion of its own and everything substantive sits in the write-up.

They are the vendor of the tool being evaluated, and the figures come from their own write-up rather than an independent test. That makes the 79% baseline and 93% zero-shot pair internally consistent, since the same team chose the corpus and the scoring, but it also means the methodology has not been checked by anyone with no stake in the result.

The denominator. A 100-call sample of production coding-agent traffic is real rather than invented, which is the write-up strongest point, but a benign-heavy corpus makes a high accuracy cheap to earn. Neither the thread nor the entry copy reports precision or a false-positive rate, and that is the number that decides whether a gate is safe to automate.