Docs
CommunityJev Benchmarks & Evaluations
BABE
A social-jev dataset for BABE: labelled rows pair query types, polarity, messages and integer targets across 362 train, 39 validation and 44 test examples, with an original config of 3,709 span-annotated rows.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- SocialJev
- Last verified
- Oct 2026
Highlights
- Labelled config: 362 train, 39 validation and 44 test rows with query_types, polarity, messages and integer targets.
- Original config: 3,709 rows, about 90 MB, with span-level label records and vote distributions.
- Original metadata carries selected_cohort, selected_kind and source_position fields.
- Annotations list kind, release, position and row values; unalignable highlights keep an excerpt and a reason.
- Sized 1K–10K examples; parquet and text, tagged with the datasets, pandas and polars libraries.
Watch out
Card reviewed 2026-10-17; the card is schema-only — no prose, no stated licence and no provenance — so the labelling process behind query_types and the integer targets is unverified, and the unalignable_highlights field suggests some annotations could not be tied to spans.
Questions about this entry
Short answers drawn from the linked project or page.
A social-jev dataset repo for BABE with a labelled config of 362 train, 39 validation and 44 test typed-decision rows and an original config of 3,709 rows of span-annotated text.
id, document_id, model, query_types, polarity, role/content messages and integer targets, matching the shape used by the other social-jev labelled configs.
3,709 text rows with label records containing id, label, definition, aggregation, value/count/proportion distributions and spans, plus metadata with selected_cohort, selected_kind and source_position.
The original config lists unalignable_highlights with an excerpt and a reason per item, marking source highlights that could not be aligned to a span in the stored text.
No licence is stated on the card; only size, format and library tags appear, so check the upstream BABE terms before reuse.
The labelled config follows a state-plus-typed-questions shape with per-option targets, but the card carries no prose explaining the labels, so it is best treated as raw material for your own validation.
Everything about semantics is undocumented: the card is a schema and split listing only, with no task description, provenance trace or licence statement to ground a stronger claim.
More Jev resources like this
A social-jev dataset for BASIL: the labelled config pairs query types, polarity and messages with integer targets across 201 train, 24 validation and 25 test rows, while the original config keeps 300 span-annotated source rows.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
A social-jev dataset for MFRC: labelled rows pair query types, polarity, messages and targets across 312 train, 39 validation and 39 test examples, and the original config keeps 17,844 rows with subreddit and vote metadata.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptClassifies 1,000 recent arXiv AI papers into 24 topics with one Jev decision each, then checks those labels against an LLM judge on a seeded sample of 100. Ships three scripts and a static topic-map page.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-papersPython
stas4000/jev-papersPythonNotebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter NotebookSammyliu's audio decision corpus for reinforcement learning and evaluation: causal audio windows paired with runtime action questions, each receiving a categorical label distribution. Train holds 168,494 decisions over 71,952 source windows; test, dev, calibration and an external diagnostic add about 122,600 more.
Jev Benchmarks & EvaluationsDocs#community#evaluation#audio
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.