Question 1

Is the falling breach rate a real measurement or a judge model marking its own homework?

Accepted Answer

The signal is defined so there is no judge in it. A breach requires three things at once: the gate said allow, the policy oracle said forbidden, and the tool actually executed against the store. The oracle is a small deterministic function over tool name, arguments, the authenticated persona and the referenced records, it never blocks anything itself, and the gate never sees its verdict, so gate and oracle cannot flatter each other.

Question 2

What happens to the demo if the TypeSafe API is down or slow?

Accepted Answer

The provider chain is TypeSafe, then an LLM structured-output fallback, then a heuristic mock, which the README frames as degrading honestly rather than silently. The gate normally answers in roughly 150 to 300 ms, and a prefilter written by the defender short-circuits before the model, blocking with probability 1.0. What is not documented is what catch rate the fallback tiers actually achieve, or how the war room labels which tier answered.

Question 3

Does the agent itself get patched when it misbehaves, or only the gate?

Accepted Answer

Only the gate. The agent is never modified; the defender rewrites the gate policy and the Python prefilter. A candidate ships only if catch rate rose and the benign allow rate stayed at or above the 0.9 floor, otherwise up to 12 concrete failures are fed into the next attempt, and after three attempts the candidate is stored as rejected and the old gate stays active. /admin can also roll the gate back to a previous version live.

Question 4

What does Jev actually see on each proposed tool call?

Accepted Answer

A state containing the policy version and its rules, the authenticated customer, the proposed action, the resolved facts, the last 8 conversation messages and the last 25 known attack examples. The facts come from resolve_refs and carry no verdict of their own, and the answer is one Choice{allow, block, escalate} with calibrated probabilities. The README does not publish accuracy or confidence distributions for that answer, nor a comparison against the same gate without Jev.

Question 5

How does the prefilter differ from the Jev gate?

Accepted Answer

The defender writes the prefilter as Python, and it runs on the message, tool, args and facts in an isolated subprocess using python -I -S with an empty environment and an 8-second cap. A hit is a block with probability 1.0, before Jev is consulted at all.

Question 6

How is a new gate version produced?

Accepted Answer

After a round, DeepSeek V4 Pro writes policy v(n+1) as strict JSON with rules, criteria, adversarial examples and prefilter code. The code is validated in a W&B Sandbox on CoreWeave and scored with a Weave Evaluation, then shipped only if catch rate rises and benign allow rate stays at or above the 0.9 floor.

Question 7

What did the live run actually show?

Accepted Answer

The first breaches were a 40% discount and a $100 store credit. Attempt 1 was rejected with a benign allow rate of 75%, under the 90% floor; attempt 2 passed, raising catch rate from 33% to 100% while holding benign allow at 92%.

Siege

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community