AppliedMachineLearning-Lab/jev-benchmarking
Organization repository on GitHub
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Evaluating and Benchmarking the System One Model Jev (arXiv 2609.37647)
A zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.
- Stars
- 6
- Language
- Python
- License
- MIT
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- Evaluates jev-1.13.0 zero-shot on 37 datasets with one frozen template per dataset and full evaluation splits.
- The full run covers 346,009 requests for under USD 10.
- Reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC; 86.7% on Belebele across 122 languages.
- Beats Qwen3.8-27B on 27 of 37 datasets and Gemma-4-E4B on all 37.
- Choice probabilities are well calibrated and support selective prediction; binary probabilities rank well but need tuned thresholds.
Watch out
arXiv preprint posted September 29, 2026 (cs.CL, cs.AI) by Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa. It is an outside benchmark of a vendor model; the abstract notes all three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based quality judgments, and thresholds tuned on training data may not transfer.
Questions about this entry
Short answers drawn from the linked project or page.
A commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed options, a position on a rubric, or the probability that a statement is true.
37 datasets spanning classification, routing, natural language inference, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring, totaling 346,009 requests for under USD 10.
It reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC, and 86.7% on Belebele across 122 languages.
Jev beats Qwen3.8-27B on 27 of 37 datasets, with none of Qwen's nine leads outside the bootstrap intervals, and beats Gemma-4-E4B on all 37.
On low-resource languages, fine-grained or noisy labels, and rubric-based quality judgments.
No. Rotating the options leaves Jev's accuracy unchanged, and withholding the question drops it to near chance, which rules out shallow memorization but not memorized question-answer pairs.
Binary probabilities rank well but are poorly placed relative to a fixed 0.5 threshold; thresholds tuned on training data raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: 100,000 viral posts in 20.4 seconds
X: SEO and GEO fixes, 90% cheaper
X: Hard-coded rules moved to Jev
YouTube: Joe Maddalone tries Jev as a routing and filtering step, with a transcript-analyzer demo.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
VictorYXL/GraphDecisionBenchPython
VictorYXL/GraphDecisionBenchPythonHATEDECIDE evaluates six decision-model configurations on four hate-speech datasets against specialized, zero-shot, commercial, and supervised baselines. The best hosted model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.