Question 1

Can I actually run this against Jev today?

Accepted Answer

Not as tested. jevBackend is a documented contract stub for POST /v1/systemone, written while early-access keys had not shipped; the README says to adjust field mapping on the first live call. What was live-tested is llmJsonBackend against a GLM endpoint using JSON forcing, where judgments were sensible on first contact. The interface, not the Jev binding, is the stable part.

Question 2

Seven to fifteen seconds per batched call - what does that do to my agent loop?

Accepted Answer

It is the reason the Jev backend exists, and the swap is not measured. The llmJsonBackend figure is quoted as about 7-15s per batched call; the Jev path is expected at 70-500ms at roughly $0.042/M input tokens, but that is an expectation stated in the README, not a benchmark. With gates firing before shell commands and after each step, you would be trading latency for judgment coverage.

Question 3

What stops a drifting LLM judge from blocking correct work?

Accepted Answer

Conservative defaults and a replay log. The README recommends thresholds at or above 0.9 to require and at or below 0.2 to forbid, warns that LLM judges drift even when first-contact judgments look sensible, and points at the decision log: every harness.run appends the gate, judgments, action and stateHash to logDir/decisions.jsonl so you can replay old PRs whose outcomes you know and tune thresholds before trusting any backend in production.

Question 4

How much of my harness do I have to change to adopt this?

Accepted Answer

You vendor the src/ folder into your TypeScript agent harness - there is no package to install - and implement JudgeBackend, which is one method, if neither llmJsonBackend nor jevBackend fits. The surface you call is nine gates: prd-gate, coverage-gate, plan-gate, pre-exec-gate, diff-gate, done-gate, route-gate, ci-triage-gate and release-gate, and the README's framing is that code owns side effects while judges only decide.

Question 5

What is a gate, and what happens when one fails?

Accepted Answer

Nine named gates map phases from PRD intake to release, each resolving to a human approval, a retry loop, or nothing. On failure prd-gate, pre-exec-gate, route-gate and release-gate escalate to a human, while coverage-gate, plan-gate, diff-gate, done-gate and ci-triage-gate loop. The README's framing is that frontier models generate, judges decide, and code owns side effects.

Question 6

Can I run anything without a Jev or GLM key?

Accepted Answer

The deterministic suite, yes. npm test covers the policy engine, the parser and the gates against a fake backend with no keys and no network, while npm run test:live makes three real GLM calls that check it blocks rm -rf, passes grep and loops on an incomplete diff. If neither bundled backend fits, JudgeBackend is one method you implement yourself.

Question 7

What does done-gate check at completion?

Accepted Answer

It asks one Noul per acceptance criterion in a single batch against the diff, and on failure returns a loop action carrying the failing criteria, which the harness feeds back through agent.continueWith(done.failures). The quickstart runs it with a list such as adds min-w-0 to the select trigger and updates the snapshot, so each acceptance criterion gets its own scored judgment.

Zcode Jev

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community