Docs
CommunityJev Benchmarks & Evaluations
BASIL
A social-jev dataset for BASIL: the labelled config pairs query types, polarity and messages with integer targets across 201 train, 24 validation and 25 test rows, while the original config keeps 300 span-annotated source rows.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- SocialJev
- Last verified
- Oct 2026
Highlights
- Two configs: labelled with 201 train, 24 validation and 25 test rows, and original with 300 rows at about 11.9 MB.
- Labelled rows carry id, document_id, model, query_types, polarity, message role/content pairs and integer targets.
- Original rows keep text with label records — id, label, definition, aggregation, distribution, span — plus article metadata.
- Article metadata includes article_id, title, article_url, source split, source version, source file and an annotations JSON.
- Size category is under 1K examples; parquet and text modality with the datasets, pandas and polars libraries tagged.
Watch out
Card reviewed 2026-10-17; the repository card is a dataset_info schema only — no prose description, no stated licence and no provenance notes — so label semantics and reuse terms are undocumented. All counts come from the metadata block.
Questions about this entry
Short answers drawn from the linked project or page.
A small social-jev dataset repo for BASIL with two configs: a labelled set of 201 train, 24 validation and 25 test rows in a typed-decision shape, and an original config of 300 span-annotated rows.
id, document_id, model, query_types, polarity, messages with role and content pairs, and targets as integer lists; the split counts are 201 train, 24 validation and 25 test rows.
300 rows of text with label records — id, label, definition, aggregation, distribution and span — plus article metadata such as article_id, title, article_url, source split, version and file, and an annotations JSON.
No licence appears in the card or the metadata; only size, format and library tags are given, so check the upstream BASIL terms before reusing anything from this repository.
The labelled shape pairs a document with query_types and integer targets, a typed-decision form close to the System One request shape, but the card never explains how the targets were produced.
The card tags the datasets library and parquet paths, so both configs load by name — labelled with train, val and test splits and original with a single train split — through the standard datasets tooling.
The card carries no prose at all: no task description, no provenance, no licence and no baselines, so the meaning of query_types, polarity and the integer targets must be inferred from field names.
More Jev resources like this
A social-jev dataset for MFRC: labelled rows pair query types, polarity, messages and targets across 312 train, 39 validation and 39 test examples, and the original config keeps 17,844 rows with subreddit and vote metadata.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
A social-jev dataset for BABE: labelled rows pair query types, polarity, messages and integer targets across 362 train, 39 validation and 44 test examples, with an original config of 3,709 span-annotated rows.
Jev Benchmarks & EvaluationsDocs#community#evaluation#classification
Community
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScriptClassifies 1,000 recent arXiv AI papers into 24 topics with one Jev decision each, then checks those labels against an LLM judge on a seeded sample of 100. Ships three scripts and a static topic-map page.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-papersPython
stas4000/jev-papersPythonNotebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter NotebookSammyliu's audio decision corpus for reinforcement learning and evaluation: causal audio windows paired with runtime action questions, each receiving a categorical label distribution. Train holds 168,494 decisions over 71,952 source windows; test, dev, calibration and an external diagnostic add about 122,600 more.
Jev Benchmarks & EvaluationsDocs#community#evaluation#audio
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.