Docs
CommunityJev Benchmarks & Evaluations
JevAlt Training Data
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- mertkayacs
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 26,043 rows: English 9,855/495/286, Turkish 7,182/391/251 and German 6,846/448/289 across train, calibration and validation.
- Question types are choice, yes/no, ordered scores and abstain; each option carries a probability target built as 0.98 times the mean plus 0.02 uniform.
- Used to train Deem-4B, Karar-4B and Wähler-4B; the card reports Karar-4B at 96.8% against Kev-4B's 87.1% on Turkish from the same pipeline.
- Sources mix teacher-written scenarios, Emberwick game requests, F1–F12 fix sets for injection and negation, and converted MASSIVE, Open-Jev and PAWS-X rows.
- Rows keep per-record licence fields; held-out test rows live in the separate jevalt-bench dataset.
Quickstart
python
from datasets import load_dataset
data = load_dataset("mertkayacs/jevalt-data", "tr")
print(data["train"][0])Watch out
Card reviewed 2026-10-17; the dataset-level licence is 'other' because rows carry mixed permissive terms, so read each record's license field. Labels come from teacher models of other families, the Karar-4B 96.8% figure is the project's own result on its own pipeline, and repeated MASSIVE utterances appear across splits.
Questions about this entry
Short answers drawn from the linked project or page.
Decision-training data for small language models: 26,043 situations with typed questions and probability targets in English, Turkish and German, published by the JevAlt project and used to train Deem-4B, Karar-4B and Wähler-4B.
A state, typed questions, and targets with an answer label plus a probability per option; rows also record language, source, teacher distributions, tags and licence, and some include a reasoning trace or an unknown option for abstention.
Two teachers from a different lab than the writer label each written scenario, each seeing a shuffled option order, and the target is 0.98 times the teacher mean plus 0.02 uniform; a question stays only when both teachers and the writer agree.
English has 9,855 train / 495 calibration / 286 validation rows, Turkish 7,182 / 391 / 251 and German 6,846 / 448 / 289, with splits assigned by a keyed hash so related rows stay together.
Targeted rows for known weak spots: missing deciding facts and unknown, negation consistency, noul versus equivalent choice, dates and business days, numbers and unit conversions, long padding, prompt injection, inverted criteria and long policies.
Mixed permissive terms with a per-record license field; written rows are Apache-2.0 while converted records retain source terms such as MASSIVE, Open-Jev, PAWS-X and typed-decisions, so check each row before reuse.
That Karar-4B scores 96.8% against Kev-4B's 87.1% on Turkish decisions from the same data pipeline, a self-reported project result; the card itself notes test and training rows share a pipeline, so it is not an independent general ranking.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Both Jev 1.13 and 1.14 score 29 of 30 on 30 Turkish buyer and seller messages
More Jev resources like this
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
The training and validation data behind jaeyong2's clef-flash-jev-claude-distill fine-tune: 37,932 states and 150,827 typed questions with soft target distributions, plus a 19,025-record language pool. The card reports the fine-tune lifts the Decision Index public suite from 52.55 to 54.96. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPythonjohannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
A benchmark of general decision models that unifies 36 datasets, 58 subtasks and 11,257 examples in one test split, with choice, noul and score questions in English and Chinese and 25 configurations scored.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.