Jev vs. the right baselines

A pre-registered, independent evaluation of TypeSafe's Jev run 2026-09-18, two days after launch, and updated with a second study on 2026-09-22. The author is not affiliated with TypeSafe and had no vendor involvement or free credits; the ~$1 in API costs was paid by the author.

Jev was a strong zero-shot classifier: +7.5pp over gpt-5.4-nano on CLINC150 (paired 95% CI [+3.0, +12.5]) but 4.5pp below frontier GPT-5.6 Terra (CI [+2.0, +7.5] in Terra's favor). Both experiments were pre-registered with kill/go criteria, and both returned AMBIGUOUS.

Recorded median call duration was about 2.2x shorter for the Jev configuration on the same 30 items (0.42 s vs 0.92 s) — not the 40–200x that vendor comparisons suggest. The README notes this compares client-and-service configurations, not isolated model inference speed, and throughput under rate limiting is far worse.

Where labeled data exists: a 9 ms supervised encoder beat the frontier model by 5.8pp on Banking77 (paired CI [+2.4, +9.6]), at no per-call cost, running on a laptop.

On CLINC150 it did not beat an LLM's verbalized confidence (AUROC 0.734 vs 0.816; paired diff CI [−0.187, +0.008], which includes zero). The opposite ordering appeared on Banking77. The README states neither direction is established.

Across Jev 1.13.0 and two open-weight replicas, with 1,200 decisions per system on frozen third-party items, Jev's confidence ranked hijacked decisions above resisted ones (AUROC 0.261, 95% CI [0.190, 0.336]). The README stresses this is not a claim the gate fails in deployment: at the frozen τ = 0.85 threshold it still rejects 69% of Jev's hijacked decisions.

The first published version overstated several results, and the first correction round introduced a new error of its own; three rounds of external review (prompts and outputs in reviews/) fixed the calibration language, escalation arithmetic, missing cross-fit accuracies and a threshold-margin sensitivity that flips the sign of the headline cascade result.