Are the Korean versus English score gaps statistically real?

No. With 100 questions per cell the uncertainty is roughly 8 points, so only the Korean medical gap for Luna clears zero. Treat the rest as suggestive rather than conclusive.

Do the Korean and English rows test the same questions?

Yes. Belebele and PAWS-X use the same questions professionally translated, so those rows isolate language. The two medical sets are unrelated exams and cannot be subtracted.

Should I translate my prompts to English first?

Korean instructions scored within a point or two of English instructions on Korean content, so the digest advises against translating prompts. Reordering inputs still shifts about one answer in eight in both languages.

Why compare against Luna, and what does Jev return differently?

Jev returns a probability per option at once with no reasoning text to parse, billed on input only, at 221 ms median against 1110 ms for the reference model. That shape is also what enables the confidence analysis.

How is the Korean check sized?

100 questions per cell drawn from four public test sets, so each number carries roughly 8 points of uncertainty and the author calls it a sample check, not a benchmark.

How fast and how priced is Jev in this check?

Jev's median is 221 ms against Luna's 1110 ms, and Jev is billed on input only because it produces no output tokens, where Luna spent about 12 per answer.

Which datasets does the check use?

Belebele reading comprehension and PAWS-X paraphrase, which are the same questions professionally translated and isolate language, plus KorMedMCQA and MedQA, two unrelated medical licensing exams.

CommunityJev Practices & Patterns

Jev Korean Benchmark

Frozen 100-question-per-cell check of TypeSafe Jev on Korean versus English reading, paraphrase, and medical exam items, with recorded responses and uncertainty intervals.

See highlights

Open github.com

Stars: 6
Language: Python
Published by: Community
Author: mahlernim
Last verified: Oct 2026

mahlernim/jev-korean-benchmark

User repository on GitHub

View on GitHub

Reproducible early-access evaluation of Jev on Korean understanding and medical text, with runtime and cost evidence

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.Homepage

About this project

Can you use TypeSafe Jev on Korean text, or should you translate everything to English first? This is a small, frozen, reproducible check that tries to answer that — 100 questions per cell, drawn from four public test sets, with every response recorded. It is a sample check, not a benchmark: 100 questions give roughly ±8 points of uncertainty, so small differences here are not findings.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

Korean reading comprehension scored 96 against 97 in English on identical Belebele items.
Paraphrase judgement was the weak spot in both languages at 76 Korean and 80 English.
Includes a Luna reference point plus order-swap and confidence analysis over recorded answers.
Reports speed, input-only billing, and per-option probabilities rather than generated text.

Watch out

100 questions per cell means wide uncertainty bands, and the two medical exams are different tests rather than a language comparison. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.