Article
CommunityJev Benchmarks & Evaluations
How Well Do Decision Models Transfer to Vietnamese? An Empirical Study of Text Classification
Empirical study evaluating four specialized decision models on zero-shot text classification in Vietnamese across 13 classification settings and six task groups, testing English, Vietnamese, and bilingual label descriptions plus teencode, typo, and diacritic perturbations.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Evaluates four specialized decision models across 13 classification settings spanning six task groups.
- Tests English, Vietnamese, and bilingual label descriptions for zero-shot text classification.
- JEV ranks first in 11 classification settings when English label descriptions are used.
- Diacritic removal reduces mean macro-F1 by 9.51-15.47 points across model configurations.
- Bilingual label descriptions give no consistent improvement over single-language descriptions.
Questions about this entry
Short answers drawn from the linked project or page.
It evaluates four specialized decision models on zero-shot text classification in Vietnamese, covering 13 classification settings across six task groups. It measures overall performance, sensitivity to label-description language, and robustness to common Vietnamese linguistic variations.
The study compares English, Vietnamese, and bilingual label descriptions. Effects varied across models and datasets, and bilingual descriptions did not provide a consistent improvement over the other choices.
Under English label descriptions, JEV achieves the strongest overall performance and ranks first in 11 of the classification settings examined.
Diacritic removal causes the most substantial performance degradation, reducing mean macro-F1 by 9.51 to 15.47 points across model configurations. This is considerably more than teencode or typographical errors.
The paper reports mean macro-F1, noting that diacritic removal lowers it by 9.51 to 15.47 points across model configurations.
The provided digest describes an empirical study and does not mention code, datasets, installation steps, or a runnable benchmark release. It is best treated as a paper write-up.
The study focuses on zero-shot text classification in Vietnamese across six task groups. It is not presented as a general evaluation of Jev behavior in other languages, tasks, or production deployment settings.
More Jev resources like this
arXiv study applying Jev and Clef-Flash to manufacturing defect detection with as few as two reference images per class, including agentic refinement of text descriptions and probability calibration. Ships replication code, an R decisions package, and a no-code web app.
Jev Benchmarks & EvaluationsArticle#arxiv#calibration#classification
Community
Case study evaluating Jev 1.13.0 on time-series decision tasks, including classification, event type prediction, oil temperature direction, and stock movement, converting continuous forecasts into discrete decisions and scoring accuracy and AUC.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv paper introducing PrimaCons, a four-axis benchmark measuring whether Jev gives consistent decisions when the same semantic state is expressed as different typed query primitives, plus the Cross-Primitive Residual Calibrator (CPRC) proposed to reduce that inconsistency.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study evaluating Jev as a decision reader of truncated LLM reasoning across seven reasoners and five multiple-choice benchmarks, comparing accuracy and estimated cost against forced answering at budgets from 500 to 16,000 tokens.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
arXiv study comparing Jev 1.13 with Clef, Microsoft-Decision-1, Laya, GPT-6 Luna and GPT-6 Sol on 172 rheumatology exam questions and 70 diagnostic vignettes, reporting accuracy, calibration and language effects.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.