Jev Guardrails

On 51 labelled cases Jev scored 48/51 with 1 false refusal and 2 misses against 46/51 with 0 false refusals and 5 misses for the LLM. Excluding content-filter escalation artefacts, genuine errors were 3 versus 3.

The LLM build samples 4 to 8 of 25 rules because full coverage costs more than the reply, while Jev asks all 25 in one request. Forced full coverage is the honest comparison the digest reports.

Jev spreads 0.44 to 1.00 across the same set, enabling threshold routing, while the LLM returned 0.96 to 0.99 on nearly every case so middle-branch thresholds never fired.

Install requirements, copy .env.example, add Azure and TypeSafe keys, then run evals/run_scope.py. Offline, tests/test_offline.py and demo_guardrails.py run the layer without model calls.

A multi-agent carrier support assistant: a generalized triage agent that reasons about intent rather than keywords, routing to six specialist desks over 43 mock tools. Handoffs are tools returning a Command, so a transfer re-enters the graph at the target desk inside the same turn with full history, and active_agent is checkpointed so the customer's next message goes straight back to the desk handling them.

With all 25 rules asked every turn, Jev runs at roughly 200 ms and $0.000125, about $125 per million reviews, against roughly 5,500 ms and $0.0058, about $5,800 per million, for the LLM build. That is 27 to 30 times faster and 45 to 47 times cheaper at identical coverage. A single scope check has the same shape: 451 ms against 3,277 ms, $60 against $1,122 per million.

Because each soft rule already carried violates and does_not_violate text, which is exactly NoulCriteria(true=, false=). NoulAnswer.noul is then the probability the rule was broken, a number you can threshold, which is what the LLM build cannot give you. Two things follow: the 4-to-8 rule sampling disappears, and the desk system prompt drops from 8,694 chars to 3,004 because it no longer carries rule text.