CommunityJev Benchmarks & Evaluations
Korean JEV Benchmark
25,418 Korean text-only evaluation items from 9 public datasets and 22 subsets, normalized to finite-choice decisions, with baseline results for ten model configurations.
- Format
- Docs
- Published by
- Community
- Author
- DaehanKim
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 25,418 Korean items from KLUE, KoBEST, Belebele, FunctionChat-Bench, APEACH, Korean Hate Speech/BEEP, SIB-200, EnterpriseOps-KO Policy and KD-Lawset-KO.
- Baselines cover ten configurations; Gemma 4 31B leads Korean accuracy at 94.40% and hosted JEV 1.13 leads English at 82.50%.
- Decision 2.0 Vega 27B reaches 92.38% Korean and 64.59% English with the best order macro of 85.31%.
- One evaluation split with 2–11 choices per item (supported scope up to 26) and no training or calibration split.
- Accuracy uses 25,418 Korean items plus 12,032 English MMLU-Pro items, scored as zero-shot direct decisions without reasoning.
Watch out
Licences are source-dependent: CC BY-SA 4.0, CC BY 4.0 and Apache-2.0, with no CC BY-NC 4.0 data included, and the repository-level tag is 'other'. Labels were audited with GPT-6 Luna and Codex adjudication; the card calls the release a curated derivative of public datasets, not an independent human annotation study or a guaranteed uncontaminated test set.
Questions about this entry
Short answers drawn from the linked project or page.
25,418 Korean text-only evaluation items drawn from 9 public datasets and 22 subsets, normalized to finite-choice decisions such as A, B or C.
Inference, comprehension, commonsense, classification, policy decisions, tool selection and questions about supplied statutes, drawn from sources including KLUE, KoBEST, FunctionChat-Bench and KD-Lawset-KO.
Local models ran on one RTX PRO 6000 Blackwell Server 96GB GPU with 256 warmup requests per timed pass, while hosted JEV was reached through OpenRouter, scoring 88.93% Korean and 82.50% English accuracy.
Gemma 4 31B leads Korean accuracy at 94.40%, Decision 2.0 Vega 27B reaches 92.38% Korean and 64.59% English with an 85.31% order macro, and hosted JEV 1.13 leads English at 82.50%.
Score exact-match choice accuracy against answer_id using the released choice order, and keep answers, provenance and audit judgments out of model prompts; there is one evaluation split and no training or calibration split.
Source-dependent: CC BY-SA 4.0, CC BY 4.0 and Apache-2.0, with no CC BY-NC 4.0 data included; see the LICENSE file and per-source attributions under benchmark/v1/attribution.
No. Labels were audited with GPT-6 Luna and Codex text adjudication while preserving original gold labels and excluding unresolved cases; the card calls it a curated derivative rather than an independent human annotation study.
More Jev resources like this
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Harmful/benign classification and answer-choice robustness for five type-safe models: 83,183 source requests from XSTest, OR-Bench and HarmBench in original, reversed and order-only variants.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonFrom the Jev community
Posts from builders shipping with Jev right now.