CommunityJev Benchmarks & Evaluations
search_jev Unified v2
897,035 search-agent decision rows in unified JEV format across train, validation and test, covering retrieval judgment, query understanding, evidence judgment and search actions.
- Format
- Docs
- Published by
- Community
- Author
- SearchJev
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- 897,035 rows in the snapshot: 487,012 train, 159,124 validation and 250,899 test, all in one state/questions/answers JEV format.
- Eight task families, from retrieval_judgment with 18,080 test rows to query_classification with 11,461.
- sealed_evaluation supplies 112,666 test rows from sources with no training counterparts.
- Answers are soft labels with calibrated probabilities and confidence; a difficulty field from 1 to 5 supports stratified sampling.
- Line-aligned index files add source, task family, label tier and confidence metadata for filtering and weighted evaluation.
Watch out
No licence is declared on the card, and this is a 2026-09-29 snapshot of the v2 release line with v3.0.0 already published as a successor. Test is about 45% sealed_evaluation, validation is dominated by a few sources, and qd_quality rows carry llm_teacher weak labels the card says to downweight; index and sidecar fields are provenance, not model input.
Questions about this entry
Short answers drawn from the linked project or page.
A JEV-format dataset for search agents covering retrieval relevance and quality, evidence judgment, search action decisions, query understanding and classification, and query quality assessment, plus auxiliary Chinese understanding tasks, at 897,035 rows.
487,012 train, 159,124 validation and 250,899 test rows. Each data file has a line-aligned index file with source, task family and label-tier metadata for filtering and stratified evaluation.
Eight: retrieval_judgment, query_understanding, evidence_judgment, general_understanding, search_action, query_assessment, query_classification and sealed_evaluation, with per-family counts in DATA_REPORT.md.
The card says to stratify metrics by task family, capability or source instead of reporting overall accuracy: about 45% of test is sealed_evaluation, helpsteer3_chinese has only a dozen-odd rows, and query assessment metrics should not mix with search-decision accuracy.
Soft labels: probabilities and confidence in [0,1] calibrated by label-source reliability, usable directly as soft targets or sample weights, with discrete answers taking the choice or argmax; gold discrete labels stay in the index.
It adds metadata for retrieval/qd_quality_v1 rows keyed by sample_id for downweighting, including rows where orig_label and the teacher score conflict or a rule_fix was applied.
Train is about 64% Chinese-dominant while the search_action family and the vitaminc series are English; instructions come in multiple Chinese and English rewrites, so models should follow the instruction rather than the source.
More Jev resources like this
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript17GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPython9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPython7GitHub stars
A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonFrom the Jev community
Posts from builders shipping with Jev right now.