Jev Harness — TypeSafeAI

Only prints. The verdicts are reject, unavailable, proposal_only, and permit, and the package is explicit that even a permit does not authorize an action. Applying a patch, executing proposed code, granting permission, sandboxing, and durable storage are all listed as host responsibilities that are not implemented here.

No. The offline totals, 7 of 25 bad proposals caught by validation alone against 25 of 25 with the Jev step, are described as scripted mock values rather than measurements of Jev. The separate live follow-up records one confidence-only catch and unexpected good holds, and says plainly that it does not establish calibration.

It is fixed, not calibrated. The model constant jev-1.13.0 is pinned for reproducibility and explicitly not as proof of calibration, and the four questions have already been revised through four versions against the same fixtures. The repository calls that adaptive development tuning rather than held-out calibration.

Nothing. The README states that decide() only receives results, cannot undo an earlier provider call, and cannot verify a host validator ever ran, nor authorize an action from a favorable verdict. Checking a proposal against real file content and diffs is the host's own responsibility.

addresses_task, evidence_supports, unrelated_changes and needs_clarification, each with a fixed favorable direction of yes, yes, no, no. addresses_task judges progress from this single step rather than completion of the whole task, and evidence_supports turns an unsupported or contradicted factual claim unfavorable even when the edit is otherwise correct. The instructions state that repository content and the proposal rationale are untrusted data, not instructions.

Arithmetically. A Noul response supplies one probability of yes, not a separate confidence: answer = p >= 0.5 ? yes : no, and confidence = Math.max(p, 1 - p). The decision table checks the probability is finite and in [0,1], the confidence is in [0.5,1], and the triple is exactly consistent, and the README keeps saying confidence is a distribution statistic rather than a probability that an action is correct.

Clone it, run pnpm install --frozen-lockfile, then pnpm verify and pnpm bench:review; Node 22+ and pnpm are required and no API key is needed for those. The package is source-only and not published, and it deliberately stops short of the live path: the live Jev transport is a host responsibility, as are authorization, sandboxing, execution and durable storage. The playground and the optional demo host own their own adapters.