TypeSafe publishes evals, but the numbers that matter for a build decision are the ones measured on your task. A wave of independent benchmarks and field reports now covers enough ground to say what Jev is reliably good at — and where it breaks.
The broadest board
Jevals runs three boards of 300 questions each: PubMedQA for noul, Banking77 for choice, and HelpSteer2 for helpfulness scoring, for 31,500 scored decisions across seven models against human labels. Its headline finding is a price result as much as an accuracy one: on nouls, Jev is reported statistically tied with the best of six LLMs at 1/28 of the price. Jev scores 69.0 on PubMedQA, 67.8 on Banking77, and 9.2 on HelpSteer2 against GLM-5.3's 7.8.
Where question design dominates
The most useful independent result of the season is one judge call or twelve dimension scores. Across 25,174 Jev calls and 5,477 test rows for $1.43, a single direct question lost to a dimension model on every hard task: 0.837 versus 0.908 accuracy on Japanese NLI, 0.400 versus 0.911 on a 12-option ledger classification, and 1.5% versus 37.2% flag rates on hard-benign guardrail text. The same lesson shows up in amankumar's 16,000-call run, which found a whole-document read scoring 68.6% with no prompt fixing it, and in lindfors.no's early-access trial, where rewriting short questions with careful qualifiers cut agreement from 0.89 to 0.86 and raised calibration error from 0.040 to 0.116.
What the accuracy spread looks like
Independent runs converge on a familiar shape: strong on classification with a clean label space, weaker where categories overlap. The Jev Capability Atlas collects both sides, including a suite where a typo in an option produced a 0.90-confidence wrong answer that disappeared once the typo was fixed, and cited third-party work where an emotion task reported 0.819 mean confidence at 48% accuracy. amankumar has Jev leading Enron spam at 98.7, SST-2 at 95.7, and AG News at 91.3, while losing Banking77 at 76.0 against 81.7 for gpt-5.6-luna — the same benchmark where Jevals reports 67.8 for Jev.
Latency, cost, and tails
The operational numbers are consistent across reports. nearhere.events measured 0.58 s median and $0.043 per 1,000 decisions against $0.370 for Mistral Small 4 and $2.496 for Gemini Flash-Lite, matching 48 of 50 expected decisions and rejecting none of the 13 valid events. lindfors.no answered eleven questions across 24 documents for half a cent total at a 0.32 s median. And amankumar documents the tail: at 100 in-flight calls, a few per thousand took 10 to 35 seconds, so timeouts belong in the design.
Calibration behaves
Where it is measured carefully, the probabilities hold. lindfors.no found 0% yes in the 0.0–0.1 bin and 98% in the 0.9–1.0 bin across 192 judgments — slightly underconfident, which is the safe direction. Jevals and the Capability Atlas both note the failure mode to watch for: tasks where categories overlap can produce confident errors.
What to measure on your own task
- Fit first. Run a few hundred labelled rows before committing, as every one of these reports did.
- Design the question, not just the prompt. Dimension questions and careful qualifiers beat one broad question on hard tasks.
- Set thresholds from data. amankumar suggests putting the drop line at half the lowest yes-probability any known positive received, tuned on one fold and checked on another.
- Watch the tail, not the median. Budget for timeouts and retries at concurrency.
- Check calibration in bins, not on average. A mean confidence number hides the bins where it fails.
The boards that score open models
A new layer of independent evaluation sits on top of this picture. The Typed Decision Leaderboard scores 2,018 answer-verifier items with the same labels for every entrant, each run on its own maker's code, and resolves the top of the table with paired bootstrap intervals — three systems tie within the noise, and on a held-out model's answers one open judge came out ahead of JEV with a confidence interval that excludes zero. The S1MB board spreads roughly 100 models across 100+ specialized tasks and leaves failed measurements visible instead of imputing them.
The boards are also where the benchmark's semantics get argued. The typed-decisions dataset reports Jev at 0.727 accuracy but KL 1.442, because it puts nearly all probability on one answer, and its ceiling is teacher self-agreement at about 0.735 rather than correctness. The community Jev Decision Index adds a vision board where JEV-27B-VL ranks first of 20.
Reading a model card's numbers
Four questions sort a card's claims as quickly as they sort a leaderboard's rows:
- Was the comparison fine-tuned against fine-tuned, or fine-tuned against zero-shot? Laya's 0.766 belongs to a checkpoint trained on the benchmark's own split; its base checkpoint scores 0.362 on the same decisions.
- Who ran the Jev reference? Most cards cite TypeSafe's published numbers rather than a rerun; AutoTrust's independently scored comparison is the exception that states both.
- Is the headline accuracy or calibration? Jev's published soft accuracy (0.580) beats Laya's fine-tune (0.471) even where the argmax accuracy loses, and calibration error is the harder number to move.
- What is the ceiling? Teacher self-agreement, majority class, or the benchmark's own construction set the maximum, and a score above it means teacher quirks were learned.
The practices category collects these studies alongside the pattern guides.
