CommunityJev Benchmarks & Evaluations
JevEmbed Data
1,601,157 training and 66,482 test decision questions for JevEmbed, covering choice, score and noul types across ten Parquet shards of at most 200,000 rows each.
- Format
- Docs
- Published by
- Community
- Author
- HIT-TMG
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- 1,601,157 training and 66,482 test questions, where the published test split is the source corpus's validation split.
- Each row carries request_json and answers_json as JSON strings, plus a question_type of choice, score or noul.
- JevEmbed compares question and candidate embeddings; chosen options, probabilities and expected scores all serve as targets.
- The trainer reads JSONL, and the card shows training KaLM v2.5 with max_input_tokens set to 2048.
- Input lengths were checked with the KaLM tokenizer, and the card says to check them again for a different model.
Watch out
This dataset has mixed, source-dependent licences (the hub tags include apache-2.0, cc0-1.0 and cc-by-4.0), so check each row's license field and the source report before reuse or redistribution. The 66,482-row test split is the source corpus's validation split rather than a fresh panel, answers are training labels that must stay out of encoder input, and length checks used the KaLM tokenizer specifically.
Questions about this entry
Short answers drawn from the linked project or page.
1,601,157 training and 66,482 test questions for JevEmbed, where each row is one decision with a state, a typed decision question and a reference answer.
The published test split is the source corpus's validation split; the data also records each row's original_split, which reads either train or validation.
choice, score and noul: choice answers name one option or give per-option probabilities, score uses ordered levels, level probabilities or an expected score, and noul gives a yes probability between 0 and 1.
It runs one encoder over the question and its candidates, compares their embeddings, then learns from the answer: cosine similarity plus softmax for choice and level-based score, squared error for numeric score, and a yes/no loss for noul.
Convert the Parquet rows to JSONL, copy configs/training/lora.yaml to lora-2048.yaml, set max_input_tokens to 2048, and train KaLM v2.5 with the jevembed.training module; the training guide covers other models as well.
The dataset has mixed licences: the card says to check each row's license field and the source report before reuse or redistribution, and the hub tags include Apache-2.0, CC0-1.0 and CC BY 4.0.
The answer and source fields. request_json supplies the state and decision question, while answers_json is the training label rather than model input.
More Jev resources like this
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
897,035 search-agent decision rows in unified JEV format across train, validation and test, covering retrieval judgment, query understanding, evidence judgment and search actions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript9GitHub stars
A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPython7GitHub stars
A benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonFrom the Jev community
Posts from builders shipping with Jev right now.