Article
CommunityJev Benchmarks & Evaluations
When a Judgment Layer's Self-Reported Fields Lie
Perry Link's Zenodo preprint compares three judgment layers - a local Laya judge, TypeSafe Jev and DeepSeek-V4.1-Flash - on the same 120-checkpoint task, reporting that cost is not the binding constraint, that p50 latency spans about 25-32x, and that the judges' self-reported fields fail in clustered ways.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- Perry Link
- Last verified
- Oct 2026
Highlights
- Measured per-call costs: Jev $0.0000146 to $0.0002406 over 8 direct-route points; DeepSeek-V4.1-Flash $0.0000326 to $0.00009645; Laya near zero marginal cost.
- p50 latency spans 25-32x: Laya 37.4 ms, the frontier model 671 ms, Jev 1.19 s in one n=20 run (pooled median 1.07 s).
- Main finding: self-reported fields fail; when a candidate value never appears in the input, the judge still averages P(true) = 0.5643 (n=220).
- Six phenomena were reproduced on real calls, including a truncation flag wrong in both directions and two verdict words that are unreachable under real inputs.
- Calibration correction included: REL 0.0556 / RES 0.0397 / UNC 0.2400 and AUC 0.7136; resolution exists, but Brier 0.2571 is worse than a constant's 0.2400.
Watch out
Copy reviewed against the Zenodo abstract and visible page text on 2026-10-15; the fetched page was truncated mid-abstract, so the third and fourth claims, the appendices and the artifacts were not readable. It is a self-published v2 preprint that flags its own corrections and re-measurements, and every cost, latency and calibration figure is the author's measurement on this single harness.
Questions about this entry
Short answers drawn from the linked project or page.
Three judgment layers on the same items and task: Laya, a local non-autoregressive judge with near-zero marginal cost; TypeSafe Jev, a remote typed-decision service; and DeepSeek-V4.1-Flash as a frontier autoregressive model, sampled through an API.
That cost is not the binding constraint. Jev measured $0.0000146 to $0.0002406 per call over 8 direct-route points, rising with input size; the frontier model $0.0000326 to $0.00009645. At larger states the order-of-magnitude gap disappears, and 'absolutely small' holds only at short state sizes.
Latency and the usable state window. p50 latency spans about 25-32x (37.4 ms, 671 ms, 1.19 s), and the remote service was measured at 15,002 characters and beyond where the local English model clamps at 512 tokens, about 3,082 characters.
That the judges' self-reported fields are not trustworthy. Seven phenomena were reproduced, six from real calls: a truncation flag that is wrong in both directions, 'fits: true' alongside an already-truncated input, and two verdict words (conflicted and undecided) that are unreachable under real inputs.
When the candidate value never appears in the input, the judge averaged P(true) = 0.5643 (n=220), reading 'not stated' as support. A Murphy decomposition gives REL 0.0556, RES 0.0397, UNC 0.2400 and AUC 0.7136: resolution genuinely exists, but Brier 0.2571 is worse than a constant predictor's 0.2400.
Yes, and the abstract says so: it reports four claims, one of which its own data refutes, and it carries an eighth-round re-measurement warning on the 2.54x figure, which becomes 2.32x and falls at the low effort level rather than at max.
It is a Zenodo preprint, version v2, published 22 September 2026 by Perry Link. It is self-published rather than peer-reviewed, and its own latency caveats - only one run has an artifact behind it, so no run-to-run range is reported - are stated inside the abstract itself.
More Jev resources like this
Benchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA Convex page for comparing models that answer verified multiple-choice questions about Convex, scoring them with source-weighted results and showing latency, cost and inspectable outcomes alongside the scores.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
A worked example of using Jev as an LLM judge: ten synthetic frozen refund-support traces are scored by three yes/no criteria and one ordered helpfulness rubric in a single Jev request and recorded as a Comet Opik experiment.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
patchy631/jev-as-judgePython
patchy631/jev-as-judgePythonA fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonA reproducible lab measuring how several coding agents coordinate on one repository, plus Médula, a kernel that asks TypeSafe Jev before every write whether a change collides with what the other agents are doing.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
joaquinruiz/medulaPython
joaquinruiz/medulaPythonHead-to-head benchmark for System One-style classification models across choice, noul and score questions. Jev led both hash-locked suites at 0.974 micro accuracy on v1 (8 tasks, 78 cases) and 0.964 on v2 (49 tasks, 866 cases), ahead of local Von 1.1, GLiNER2 and Laya.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
jabr/classifier-benchmarkPython
jabr/classifier-benchmarkPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.