Question 1

Are those precision and recall numbers measured, or lifted from a cookbook?

Accepted Answer

Measured, once, and re-runnable. eval_guardrail.py runs the injection question over deepset/prompt-injections, 662 labelled prompts with 263 injections across English and German, and the cassette from that run is committed, so python3 eval_guardrail.py re-scores offline with no key. It reports precision, recall and F1 across thresholds, ROC-AUC, ECE with a reliability table, Brier, latency percentiles, tokens and dollars per 1k requests against a regex baseline.

Question 2

Does it really block at 0.8, or is that number doing marketing work?

Accepted Answer

The shipped thresholds are 0.5 and 0.1, and the README says plainly that they come from the measured reliability table rather than the cookbook's 0.8 and 0.4. The reason is visible in that table: the model is under-confident here, so a prompt scored 0.15 is an injection nearly every time and the misses cluster in the 0.0 to 0.1 bin. Blocking at 0.8 would discard most of them.

Question 3

What happens when the guardrail call fails or the model is unreachable?

Accepted Answer

The gateway fails open: it forwards to the large backend and flags the request, and FAIL_MODE=closed switches it to rejecting instead. Requests carrying a context field trigger a second Jev call that checks groundedness as supports, contradicts or says nothing, and blocks or flags a leak with 422. Every decision is a threshold in policy.py, so nothing depends on how the model phrased an answer.

Question 4

Is this actually safe against multi-turn prompt injection?

Accepted Answer

Not on its own, and the README lists this as a deliberate simplification: only the last user message is judged. Multi-turn injection needs the transcript put into state, which this version does not do. The dataset used is single-prompt too, so the measured recall should be read as per-prompt rather than per-conversation, and the regex baseline it is compared against is a weak one by construction.

Question 5

What does one request through the gateway actually cost?

Accepted Answer

The single recorded run spent 409 tokens per call and $0.017 per 1,000 requests at list price, with latency p50 226 ms, p95 291 ms and p99 381 ms. Running with --full asks all four questions per call so you can see the real fan-out latency rather than the single-question timing.

Question 6

How is the routing decision made from those answers?

Accepted Answer

policy.py holds pure functions and owns the outcome: risk at or above 0.80 returns 403, 0.40 to 0.80 allows and flags for review, and the small tier is used only when confidence reaches 0.60, otherwise the request goes to the large backend. Any OpenAI-compatible server can sit behind either backend.

Question 7

How do I run and test it, and does it need dependencies?

Accepted Answer

None. jev.py is a 90-line stdlib client for POST /v1/systemone with record and replay cassettes keyed by request hash, so python3 test_switchboard.py runs 11 offline tests covering policy, cassette replay, gateway flow and metric math. python3 gateway.py serves on port 8080 and echoes its backend unless BACKEND_SMALL or BACKEND_LARGE is set.

Switchboard — aniruddh-krovvidi

About this project

Highlights

Questions about this entry

Other listings of this project

More Jev resources like this

From the Jev community