Docs
CommunityJev Benchmarks & Evaluations
Audio Jev RLCD v1
Sammyliu's audio decision corpus for reinforcement learning and evaluation: causal audio windows paired with runtime action questions, each receiving a categorical label distribution. Train holds 168,494 decisions over 71,952 source windows; test, dev, calibration and an external diagnostic add about 122,600 more.
- Format
- Docs
- Reading time
- 3 min
- Published by
- Community
- Author
- sammyliu
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- Train holds 168,494 decisions over 71,952 source windows; test has 82,331 decisions over 33,944 windows.
- Five targets: speech activity, speaker count (0-9 and 10+), turn completion, floor transition and next speaker.
- Compiled from Smart Turn v3.2, AMI SDM and VoxConverse at pinned immutable revisions, distributed CC-BY-4.0.
- Model-visible audio is physically cropped before any future-derived target is attached; inputs carry no transcripts.
- A 9,998-decision external_test diagnostic sits in a separate corpus root and is never a selection split.
Watch out
Card reviewed 2026-10-17; split counts and composition are the card's own tables. The corpus is a derivative compilation without endorsement by the source authors, windows within a recording overlap and are not independent people or meetings, count and duration strata are selected for coverage rather than population prevalence, and calibration applies to this selected distribution only.
Questions about this entry
Short answers drawn from the linked project or page.
A corpus of causal audio windows and runtime action questions for reinforcement learning and evaluation: audio is supplied once and each question receives a categorical label distribution over its options. It contains gold and source annotations rather than teacher probabilities or policy labels, and it is distributed as a CC-BY-4.0 derivative compilation.
Each item is a state-shaped audio window plus typed questions whose targets are probability distributions summing to one, matching the decision-model shape Jev uses. The card positions the corpus for reinforcement learning and evaluation of runtime decisions such as speech activity, turn completion and next-speaker choice, not for text generation.
Five targets: speech_activity, speaker_count with labels 0-9 and 10+, turn_complete from the supplied Smart Turn semantic endpoint annotation, floor_transition over the following two seconds, and next_speaker among the source-annotated audible voices or nobody. Every categorical target sums to one.
Smart Turn v3.2 train and test, AMI SDM and VoxConverse, all pinned to immutable revisions with attribution listed on the card. The compilation is distributed as CC-BY-4.0 with source notices preserved, and the card states it is a derivative work without endorsement by the source authors.
Train holds 168,494 decisions over 71,952 source windows (525.44 audio hours); dev has 14,873 decisions, calibration 15,425 and test 82,331, with a separate 9,998-decision external_test diagnostic. Hours sum supplied windows including overlaps, so they are not independent recording hours.
floor_transition is a future-outcome proxy, not a gold semantic completion or backchannel judgement, and next_speaker candidates come from source-annotated voices, an oracle inventory rather than unconstrained speaker discovery. Windows overlap within recordings and must not be treated as independent people or meetings.
external_test is never a selection split because its source collections can overlap the development and training collections. The card says to report natural and synthetic subsets separately, keep targets and provenance outside model context, and validate deployment calibration separately since strata were selected for coverage, not prevalence.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Both hands on a piano, in real time
More Jev resources like this
Teacher-labelled distillation pack for the one-token Qwen decision track: 10,044 rows across train, eval and calibration JSONL files, plus 1,708 labelled fixtures. Labels come from jev-1.13.0, and the card says not to train on the eval or calib splits. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
The training and validation data behind jaeyong2's clef-flash-jev-claude-distill fine-tune: 37,932 states and 150,827 typed questions with soft target distributions, plus a 19,025-record language pool. The card reports the fine-tune lifts the Decision Index public suite from 52.55 to 54.96. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#distillation
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.