Docs
CommunityJev Benchmarks & Evaluations
Typed Decisions (Deutsch)
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- johannhartmann
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- German translation of LocalLLaMA/typed-decisions at revision c806f6c, same configs, splits, columns and row ids.
- 2,217 unique strings translated; JSON keys, question names, option keys and identifiers stay English.
- Gold distributions come from the original teacher run on English text and were not regenerated.
- Translated with Claude against a glossary of all 75 question and option texts plus the 82 most frequent values.
- Automated checks confirm identical structure and all 8,000 questions accepted by Unsloth's decision parser.
Quickstart
python
from datasets import load_dataset
import json
ds = load_dataset("johannhartmann/typed-decisions-de", "customer_service", split="test")
row = ds[0]
state, questions, gold = (json.loads(row[k]) for k in ("state", "questions", "gold"))Watch out
Card reviewed 2026-10-17; the translation is machine-only with no human review, so tone or nuance shifts can loosen the fit between the German text and the soft labels, which were produced on the English text. The original leaderboard numbers do not transfer to this version, and the card says not to train on test, including labels derived from it.
Questions about this entry
Short answers drawn from the linked project or page.
A German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c, released under the same Apache-2.0 licence. Layout, configs, splits, columns and row ids match the original exactly; only the natural-language text was translated.
All natural-language values in state, questions (instructions, option descriptions, score rubrics) and factors were translated. JSON keys, question names, option keys, ids, hosts, paths, amounts and every gold or flat label column are unchanged, so gold still points at the same options.
From the original teacher run on the English text; no new labels were produced for the German text. Small shifts in tone or nuance can therefore loosen the match between the translated text and the soft target distribution.
Machine translation with Claude and no human review: 2,217 unique strings translated against a glossary covering all 75 question and option texts and the 82 most frequent values, with automated checks for coverage, preserved numbers, identical structure and parser acceptance of all 8,000 questions.
No. The card says the original leaderboard values apply to the English version and do not transfer to this translation; the German dataset is for training and evaluation on German text, not for direct score comparison with the English runs.
With the datasets library, then json.loads the state, questions and gold columns. In Unsloth Studio it is selectable as a decision-model training dataset under the all subset.
Five configs - agent_trace_observability, customer_service, invoice_processing, security_incidents and all - mirroring the original, each with train and test splits. Apache-2.0, the same as the source, and the translation is not connected to the original authors.
More Jev resources like this
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
xiq's JEV-format view of OpenJev-Data: every browser step re-expressed as state, questions and gold across eight configs covering Mind2Web, NNetNav, WebChain, two teacher runs and on-policy GLM, plus a frozen 2,346-step bench and 153 live tasks. Licences are mixed and per-row terms apply.
Jev Benchmarks & EvaluationsDocs#community#evaluation#browser-agent
Community
Twelve frozen typed-decision configs from the independent Open-Jev project, publishing controlled and mostly synthetic tasks with probability or ordinal targets: from the 113,568-row default release to the 160,797-row browser and drone expansion. CC0 1.0; English, Chinese and Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.