The first month of Jev produced an academic literature of its own: more than eighty preprints now evaluate, attack, and reproduce the typed-decision model class. This is what they collectively found. Every claim below is from a preprint, and most results are single-run — for field reports and vendor evals, read measuring-jev instead.
The broadest evaluation
Evaluating and Benchmarking the System One Model Jev runs 346,009 requests across 37 datasets for under USD 10. It reports 95–99% on IMDB, SST-2, HellaSwag and ARC, 86.7% on Belebele across 122 languages, and wins against Qwen3.8-27B on 27 of 37 datasets with no Qwen lead outside the bootstrap intervals. It also shows the shape of the failure surface: all models degrade on low-resource languages, fine-grained labels, and rubric judgments.
Accuracy is not where the fight is
Two papers argue the typed readout's edge is operational, not accuracy. JEV versus LLMs replicated seven political-science studies and found Jev matches the LLMs but has no cost advantage at OpenAI's batch prices. Fast Models, Slow Evidence built 11 agent decision points and found Jev significantly more accurate on 9 of them (+10.8 to +46.0 points) while neither model beats chance on zero-shot model routing — and it audited its own pipeline, walking back headline savings from 23.9% to 4.3% after fixing analysis errors. The early evidence audit, reviewing 28 papers from the first nine days, concludes the typed readout itself has not shown an independent accuracy advantage.
Calibration holds; coherence does not
Beyond Calibration asks whether probabilities for logically linked questions fit together: Jev's negation pairs miss summing to one by 0.064 on average, and its three single-label probabilities sum to 1.14. Calibration and coherence are different properties, and only the first is well measured. Calibrated to Whom adds that answers are highly repeatable (ICC 0.997) but persona- and language-sensitive: a Saudi persona reproduced 87% of the human Saudi-US difference in English and 62% in Arabic.
Rejection is the weak spot
On paired arithmetic problems, the rejection bottleneck paper measured 99% answer-present accuracy against 7% correct rejection — while the same model verified candidate correctness at 99% as a Noul, and a threshold tuned on separate problems raised rejection to 79%. Candidate-coverage work shows why thresholds tuned elsewhere fail: a policy calibrated on DBpedia rejected 69.3% of covered Emotion inputs.
Attacks work, and defenses are thin
Same Attack, Different Judge finds Jev the most robust judge on input moderation — eight structured-input forgeries bypassed none of its blocks, against 56% bypass for GPT-6 Astra on a why-might-someone-ask sentence — but on phishing it detects only 62% of flagged emails and forgeries bypass 19–29%. Nudge the Judge traced hijacking across 228 real Jev applications and found decision flips in 81.5%.
The model class travels
Theory-of-mind evaluation put Jev first or second across six capability groups on 11,117 questions, at 73.2% source-balanced accuracy. Indonesian regional languages raised accuracy up to 37.2 points by presenting an Indonesian translation, without expected-calibration-error growth, though error ranking degraded. And labels override definitions shows accuracy can change by 0.1511 from renaming options alone — a prompt-rendering bug, not a model limitation.
What to take from it
- Preprints, not peer review: treat every number as a hypothesis with a methodology attached.
- The readout's measured wins are cost and latency; accuracy still depends on question design.
- Rejection, coherence, order effects, and context attacks are the failure modes to test first.
- The class moves fast: open reimplementations now match or beat the hosted model on several of these same benchmarks.
