Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
CommunityJev Benchmarks & Evaluations

Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languages (arXiv 2610.14112)

A pre-registered evaluation of the commercial Jev model and open Laya-multilingual on Indonesian regional languages, using the parallel NusaX and NusaTranslation test sets. Jev is more accurate on sentences lexically closer to Indonesian, and the human Indonesian translation raises accuracy by up to 37.2 points.

See highlightsOpen arxiv.orgtnromy/evaluating-system-one-models-for-indonesian-regional-languages
Stars
0
Language
HTML
License
MIT
Format
Article
Published by
Community
Last verified
Oct 2026
Tagscommunityresearcharxivmultilingualcalibrationtranslation
Works forJev for ClassificationJev for RoutingJev for Evaluation
Used inJev for Software & DevToolsJev for Data & Analytics

Highlights

  • Compares Jev and open Laya-multilingual on NusaX sentiment (12 languages) and NusaTranslation (7 languages).
  • Jev is more often correct on sentences lexically closer to Indonesian across all three datasets.
  • Supplying the human Indonesian translation raises Jev's accuracy by up to 37.2 points, most in distant languages.
  • Expected calibration error does not grow on regional languages, but Jev's ability to rank its own errors degrades.
  • Retrained classifiers need more than 500 labels per language to match zero-shot Jev in 9 of 10 NusaX regional languages.

Watch out

arXiv preprint posted October 9, 2026 (cs.CL, cs.LG; 14 pages) by Romi Fadli, Rusdah, Wahyu Ramadani Putri Utami, and Zulfa Malika Amana. The pre-registered study covers Indonesian regional languages on sentiment and emotion tasks only, and the cascade cost comparisons are the authors' own measurements.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

Benchmarking candidate coverage and rejection policy transfer in typed decision models (arXiv 2610.03387)
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Communityluckykevvv/Decision_Model_BenchmarkPython
Evaluating and Benchmarking the System One Model Jev (arXiv 2609.37647)
6GitHub stars
A zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
CommunityAppliedMachineLearning-Lab/jev-benchmarkingPython
Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks (arXiv 2610.11978)
A comparison of Jev with 19 LLMs across 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding. Jev is competitive with frontier LLMs on knowledge and commonsense tasks, but on MathQA it trails the frontier median by 17.7 points.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
What the Jev
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Communityketa1930/what-the-jevPython
Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges (arXiv 2610.18122)
A bias audit of typed decision models used as judges, running the CALM benchmark's bias tests on eight TDMs and seven LLM judges from 2024 and 2026. Every TDM is biased in three to eight of twelve tests, and emotional tone drops a typical TDM's accuracy from 99% to 42%.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev (arXiv 2610.16655)
A pre-registered study of whether jev-1.13.0 satisfies independence of irrelevant alternatives, using 299,463 calls on intent and topic classification. Every added option shifts the log-odds between the gold and strongest wrong label by 0.20–0.52 nats RMS, 1.2 to 3 times run-to-run variation.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community

From the Jev guides

Original write-ups on what this entry is for.

What the First Month of Jev Papers Found
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
Designing for Rejection
A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.
Jev Practices & Patterns#rejection#confidence#patterns
Read article
Measuring Jev: What Independent Benchmarks Actually Found
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai