Docs
CommunityJev Benchmarks & Evaluations
Open-Jev Typed Decision Datasets
Twelve frozen typed-decision configs from the independent Open-Jev project, publishing controlled and mostly synthetic tasks with probability or ordinal targets: from the 113,568-row default release to the 160,797-row browser and drone expansion. CC0 1.0; English, Chinese and Turkish.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- pengchao1587
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- Twelve configs; the default release-v2-redistributable holds 113,568 rows across five split types.
- browser-drone-expansion-v1-redistributable is the largest at 160,797 rows; release-v2 is contained within it.
- mailroom-control-v1 holds 114,800 rows, while small controls like citation-control-v1 run 4,000 rows.
- Decisions are typed: yes/no probability, choice distribution, independent label probabilities or numeric outcomes.
- The card states these are not official TypeSafe/Jev training data and Open-Jev is not affiliated with TypeSafe.
Watch out
Card reviewed 2026-10-17; the fetched card text truncates inside the configs-overlap paragraph, so the containment relationships between all twelve configs are not fully stated. Targets are controlled synthetic reference labels rather than model predictions or evidence of general capability, and per-config licence details beyond the repository-level CC0 1.0 are not enumerated.
Questions about this entry
Short answers drawn from the linked project or page.
Twelve separate, frozen data configs from the independent Open-Jev project. Each turns a state and a question into a typed decision: a yes/no probability, a distribution over choices, independent label probabilities, or a discrete numeric or ordinal decision, with manifests and reconstruction instructions.
Probability estimation over supplied options in several forms - binary yes/no, choice distributions, independent labels per option, and numeric or ordinal answers - all delivered as controlled, mostly synthetic tasks with reference labels rather than model predictions.
From 4,000 rows for citation-control-v1 to 160,797 for browser-drone-expansion-v1-redistributable; the default release-v2-redistributable holds 113,568 rows and mailroom-control-v1 holds 114,800. Every config splits into train, calibration, validation, test and OOD.
Yes: the card states release-v2-redistributable is contained within browser-drone-expansion-v1-redistributable, and its overlap paragraph is truncated in the fetched text, so treat that relationship as described and avoid summing config totals as if they were disjoint.
No. The card states these are controlled, mostly synthetic tasks and reference labels, not official TypeSafe/Jev training data, model predictions, or evidence of general capability, and that Open-Jev is independently implemented and not affiliated with TypeSafe.
CC0-1.0 at the repository level, with the data stored as parquet and tagged English, Chinese and Turkish. The fetched text does not break row counts down by language, though the language tags cover all three.
The shapes line up: a state plus a typed question, answered with probabilities over supplied options instead of generated text. Because the targets are synthetic references, the dataset is useful for training or sanity-checking typed-decision behaviour, not for claiming deployed accuracy.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
YouTube: Dan Makes Things drives an ESP32 robot with Jev decisions.
More Jev resources like this
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptThe research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.