Article
CommunityJev Benchmarks & Evaluations
ASSAY-001 — independent verification of Jev
Jourdan Labs' pre-registered, independently rescored audit of Jev's two launch claims: calibrated on CLINC150 (ECE 0.0204) but not on Banking77 (ECE 0.0936), and zero type errors across 8,576 responses. Protocol and receipts published in full.
- Format
- Article
- Reading time
- 3 min
- Published by
- Community
- Author
- Jourdan Labs
- Last verified
- Oct 2026
Highlights
- Verdict: calibration held on CLINC150 (ECE 0.0204, accuracy 0.8812) but not on Banking77 (ECE 0.0936, accuracy 0.7977, overconfident below 0.9).
- Zero type errors in 8,576 responses; the only contract exceptions were choice picks within 0.01 of a tie (2 on Banking77, 1 on CLINC150).
- Protocol frozen 2026-09-17 (SHA-256 f8ce3f5b...) before any query; corpora pinned by revision hash, one run per corpus, no retries.
- A second party on a different base model re-scored sealed responses blind and matched every field; the run cost about fifty cents of input tokens.
- Median end-to-end latency from central Texas: 382.3 ms on Banking77 and 385.9 ms on CLINC150, p99 852.1 and 845.1 ms.
Watch out
Page text fetched 2026-10-17. The site itself scopes the result: it applies to jev-latest on 2026-09-18, Choice questions with null option descriptions, and these two corpora only - not other tasks, days or models. Both corpora are public and old and may be in Jev's training data. Amendment 2 widened the rounding tolerance after 513 responses summed to 0.99, and 4 CLINC items went unscored after a DNS failure mid-run.
Questions about this entry
Short answers drawn from the linked project or page.
An independent verification of TypeSafe Jev's calibration and type-safety claims, run by Jourdan Labs as field journal engagement 001 from Plano, Texas. The protocol was frozen before any query, and the split verdict and full receipts are published; TypeSafe had no edit or veto.
Two verbatim claims from typesafe.ai: 'Calibrated: higher confidence means higher accuracy' and 'The model never makes type errors'. They were checked on Banking77 (77 intents, n=3,080) and CLINC150 (150 intents plus out-of-scope, n=5,496), with human-labelled corpora pinned before the first query.
On CLINC150, ECE was 0.0204 and held within 2.5 points of the diagonal in every bin (accuracy 0.8812, Brier 0.1846) - a pass. On Banking77, ECE was 0.0936 and the model was overconfident in every bin below 0.9 (accuracy 0.7977, Brier 0.3088) - the claim did not hold.
Zero violations in all 8,576 responses. The only anomalies were choice-contract exceptions within 0.01 of a tie - 2 on Banking77 and 1 on CLINC150 - and 513 responses that summed to 0.99 because Jev rounds to two decimals, which Amendment 2 treated as rounding, not an error.
The protocol, claims, corpora and pass criteria were frozen with a SHA-256 hash before access; positive controls had to go red and green before Jev was scored; every request and response was logged and sealed; and a second party on a different base model re-scored the sealed responses blind, matching every field exactly.
Calibration was measured to be corpus-dependent: it held within 2.5 points on CLINC150 but was systematically overconfident on Banking77, where a stated 0.86 was right 67% of the time. The audit says it is not a statement about Jev on other tasks, corpora or days, and not a comparison with any other model.
Jourdan Labs published it on donttrustme.ai, dated 09.18.2026, with the harness, protocol, raw responses and both scorers in the receipt, and invited TypeSafe to reproduce it. The run cost about fifty cents of input tokens at TypeSafe's published price; output tokens were free.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Reddit: Localization QA with a decision model: meaning, terminology, and placeholders as separate checks.
Hacker News: Pick Verify: a single decision at the warehouse pick station catches confusable picks.
Show HN: Verifying robot picks with a decision modelRead the thread on Hacker NewsMore Jev resources like this
A fast, typed Python framework from TrustifAI for evaluating LLMs, RAG pipelines and agents with decision models: Jev by default, Microsoft Decision-1 on OpenRouter or OpenAI's Decisions API through the same backend. It returns typed scores, thresholds and status, with optional human-label calibration.
Jev Benchmarks & Evaluations#community#github#python
Community
trustifai/typed_evalsPython
trustifai/typed_evalsPythonMetamorphic coherence benchmark for typed probabilistic decision models: 240 synthetic cases and 2,240 questions checked by 50 probability laws, with released reports for 21 open models and 4 reference models.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
The public track of a multimodal agentic-judge benchmark for Jev-style models: 3,208 items from 16 benchmarks across five judge roles, scored on option probabilities rather than a top answer, under evaluation-only terms.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
0GitHub stars
Tests whether typed-decision model probabilities satisfy coherence, using logically linked questions that need no labels. On 480 negation pairs, Jev's probabilities for a label being X and not X miss summing to one by 0.064 on average.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
Node.js CLI that checks whether published Jev outputs were actually produced by Jev: it recomputes the Yurin confidence identity and the expected-score formula over a repository's artifacts and flags fixtures that violate them. No API calls, no dependencies, Node 18+.
Jev Benchmarks & Evaluations#community#calibration#verification
Community
stillmarcus24/jev-verifyJavaScript
stillmarcus24/jev-verifyJavaScriptFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.