Docs
CommunityJev Benchmarks & Evaluations
MFRC
A social-jev dataset for MFRC: labelled rows pair query types, polarity, messages and targets across 312 train, 39 validation and 39 test examples, and the original config keeps 17,844 rows with subreddit and vote metadata.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- SocialJev
- Last verified
- Oct 2026
Highlights
- Labelled config: 312 train, 39 validation and 39 test rows with query_types, polarity, messages and integer targets.
- Original config: 17,844 rows of text with label definitions, vote proportions and spans.
- Original metadata adds subreddit, bucket, position and voting_annotator_count fields.
- Assignments list per-annotator records with position, annotator, annotation and confidence.
- Sized 10K–100K examples; parquet and text, with datasets, dask and polars tagged.
Watch out
Card reviewed 2026-10-17; the card is a dataset_info schema with no prose, no stated licence and no provenance notes, so label semantics are undocumented; counts come from the metadata block, and labelled and original are two views of the same material.
Questions about this entry
Short answers drawn from the linked project or page.
A social-jev dataset repo for MFRC: a labelled config of 312 train, 39 validation and 39 test typed-decision rows, plus an original config holding 17,844 rows of source text with annotation records.
Rows carry id, document_id, model, query_types, polarity, role/content messages, integer targets and a source_revision field; the split sizes are 312 train, 39 validation and 39 test.
17,844 text rows with label records carrying id, label, definition, aggregation, distribution values with counts and proportions, plus spans when present, and metadata with subreddit, bucket, position and annotator counts.
Yes: the assignments list records position, annotator, annotation text and a confidence value per annotator, alongside the aggregated label distribution for each row.
No licence is stated anywhere on the card; only the size, format and library tags are present, so verify the upstream MFRC terms before any reuse.
It mirrors the typed-decision shape — a text state, a set of query types and per-option targets — but the card gives no prose about how the targets were labelled or what the query types mean.
Treat the schema as the only documentation: there is no task description, provenance or baseline on the card, and the source_revision field hints at an upstream snapshot that is not further described.
More Jev resources like this
A social-jev dataset for BASIL: the labelled config pairs query types, polarity and messages with integer targets across 201 train, 24 validation and 25 test rows, while the original config keeps 300 span-annotated source rows.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
A social-jev dataset for BABE: labelled rows pair query types, polarity, messages and integer targets across 362 train, 39 validation and 44 test examples, with an original config of 3,709 span-annotated rows.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptClassifies 1,000 recent arXiv AI papers into 24 topics with one Jev decision each, then checks those labels against an LLM judge on a seeded sample of 100. Ships three scripts and a static topic-map page.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-papersPython
stas4000/jev-papersPythonNotebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter NotebookSammyliu's audio decision corpus for reinforcement learning and evaluation: causal audio windows paired with runtime action questions, each receiving a categorical label distribution. Train holds 168,494 decisions over 71,952 source windows; test, dev, calibration and an external diagnostic add about 122,600 more.
Jev Benchmarks & EvaluationsDocs#community#evaluation#audio
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.