Question 1

Are those accuracy numbers measured or just asserted?

Accepted Answer

Measured, and built so you can re-check them. jev_verify runs the built-in labeled suite against the live API and refuses to report numbers it did not measure, and the same run is reproducible from the command line with any key, optionally repeated for latency stability. What it does not do is test anything outside its own 27 questions.

Question 2

Can the guard silently block or pass something it never checked?

Accepted Answer

No, by design. A free deterministic blacklist runs first on shell-like tool calls, anything it lets through goes to Jev for a risk noul, and denial above the threshold comes with an explicit reason. Every verdict is counted and inspectable through jev_guard_status, and the loop guard injects advice instead of blocking so it never silently stops a run.

Question 3

What happens if the API is down or the key is missing?

Accepted Answer

There is no mock mode and no silent fallback. Without a key the tools and guard fail with explicit setup instructions, and the guard fails open with a warning rather than pretending it checked. Each decision result carries the model, latency and token usage so any individual call can be audited after the fact.

Question 4

Is 96% on a real benchmark, or a suite the author wrote for themselves?

Accepted Answer

The latter, and worth weighing accordingly: it is a 27-question suite maintained in this repo, guard verdicts included, with the documented miss being a boundary case. The release notes are candid that an earlier tool description claimed 24 questions across 15+ cases before being corrected to match the code, which is a reason to read lib/cases.js rather than trust any headline number.

Question 5

What are the three tools the plugin adds?

Accepted Answer

jev_decision answers choice, score and noul questions against one state in a single call and returns answers, confidence, probabilities, usage, latency and estimated cost. jev_verify runs the labeled benchmark against the live API. jev_guard_status audits the auto-guard's counts, guarded tools, thresholds and budget.

Question 6

What does the loop guard do that the safety guard does not?

Accepted Answer

It watches for stalls rather than danger. The safety guard runs before a shell-like call and can deny it; the loop guard runs after execution and flags consecutive same-tool calls with long outputs, asking Jev whether the agent is semantically stalled, then injects a non-blocking advisory into the next request and applies a cooldown. It advises rather than blocks.

Question 7

What is a concrete example of the guard blocking something?

Accepted Answer

In the README's measured demo, a recursive remove-item was caught by the free deterministic rule, while the instruction to permanently wipe all staging data and delete every row from every table was intercepted by Jev at 91 percent confidence against a threshold of 0.8, before any command ran. Both verdicts are countable through jev_guard_status.

Dsh Jev Verify

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community