A decision model that works in English is not automatically a decision model that works in Indonesian, Arabic, or Chinese. The interface is the same (state in, typed answer out), but the evidence changes with the language, and so does the calibration. The useful question is not “does it support the language?” It is “where does the language change the answer, and what do I do about it?”
Three recent evaluations give a practical map.
Closeness to the high-resource language predicts errors
Does System One Transfer? compares the commercial Jev model and open Laya-multilingual on NusaX sentiment (12 languages) and NusaTranslation sentiment and emotion (7 languages). Jev was more often correct on sentences lexically closer to Indonesian across all three datasets. Supplying the human Indonesian translation raised its accuracy by up to 37.2 points, with the largest gains in distant languages.
Two details matter for design. First, expected calibration error did not grow on the regional languages, but Jev's ability to rank its own errors degraded. A model can look calibrated on average while becoming worse at telling you which cases it is likely to get wrong. That is exactly the ordering a review queue depends on.
Second, retrained classifiers needed more than 500 labels per language to match zero-shot Jev in 9 of 10 NusaX regional languages. That is a useful baseline: before collecting a large labeled set, check what the hosted model already does. The caveat is that the study covers Indonesian regional languages on sentiment and emotion tasks only, so do not generalize the label budget to every domain.
Persona and language are different variables
Calibrated to Whom? put 288,000 answers through Jev using the Values Survey Module 2013: 24 items, 12 matched Saudi and 12 matched American personas, with and without a persona, in English and Arabic, under eight request formulations.
The findings are unusually clean. Answers were highly repeatable, with an ICC of 0.997. Without a persona, the answers resembled Jev's own American personas. A Saudi persona moved answers toward the human Saudi–US difference, reproducing 87% of its size in English but 62% in Arabic. The smaller Arabic difference came from the language of the items, not the language of the persona description. Age shifted profiles about as much as nationality, and Jev was less confident in Arabic and for Saudi personas.
The design consequence is that “add a persona” is not a language fix. Persona and language are separate inputs, and the language of the instructions can move the answer even when the persona is held fixed.
Chinese is not one benchmark
Chinese-Jev is an open model trained on a 10-million-example Chinese corpus and fine-tuned for medicine, legal, and finance. After first-stage pre-training it exceeds closed-source Jev by 1.24% on general-domain tasks with a 20.3x speedup. Domain fine-tuning gives a 4.0% accuracy improvement over Jev in medicine. On specialized domains it reaches 92% of Jev's average accuracy with 17x speedup and 15 ms average latency per example.
Those numbers are self-reported against a closed model, and the project page named in the paper is not a code repository, so weights and training data cannot be checked from the digest. Treat the result as a promising open alternative, not as an independent reproduction. The broader point still holds: a language-specific model can beat a general hosted model on its home domain, and the gap may be largest in specialized vocabulary.
What to do in a multilingual deployment
Start by measuring the language slice separately. A single accuracy number across languages hides the failure mode the Indonesian study found: the model is more accurate on inputs closer to the high-resource language, and its confidence ranking degrades on the distant ones.
Then decide where translation belongs. If the state is a document in a low-resource language, you have two options: translate the state before the request, or ask the model to decide in the original language. The Indonesian study suggests translation can raise accuracy substantially, but it also introduces a translation step that can lose nuance. For high-stakes decisions, run both and compare on a labeled sample.
Keep option labels in the language of the decision. A translated state with English labels creates a mixed-language request that may behave differently from either pure form. The advanced structure guide allows option descriptions and criteria as structured JSON, which helps you bind a label to a definition in the same language as the evidence.
Finally, re-calibrate per language. The confidence semantics are not language-neutral just because the API returns a number. A threshold tuned on English may over- or under-escalate on Arabic, and the calibration study shows that confidence can shift with the language of the items even when the underlying decision is stable.
A short checklist
- Measure accuracy and calibration separately for each language.
- Include a “lexically distant” slice in the eval, not only translated duplicates.
- Test whether translating the state helps before building a translation pipeline.
- Keep option labels and criteria in the language of the evidence.
- Re-tune gates per language; do not import an English threshold.
- Check error-ranking quality, not only ECE, because review queues depend on ordering.
- Treat language-specific open models as candidates, and verify their claims independently.
Multilingual support is not a checkbox. It is a set of distribution shifts you can measure, and the measurements tell you where the model needs help rather than where the marketing says it is ready.
