Docs
CommunityJev Benchmarks & Evaluations
JobFit JevBench
A CV/job fit-scoring benchmark from gw0: 250 synthetic CVs, 404 job postings and 5,000 pairs, each judged on 17 typed questions with a score and a confidence, built to train and evaluate the Jev-like jobfit-model.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- gw0
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- 250 synthetic CVs, 404 job postings and 5,000 CV/job pairs, each judged on 17 questions with a 0–1 score and a confidence.
- 13 pairwise questions plus 2 CV-level and 2 job-level questions; the five score levels map to 0, 0.25, 0.5, 0.75 and 1.
- Labels come from claude-sonnet-5, with a second rater, claude-opus-5-5, on 300 pairs, 25 CVs and 87 jobs for agreement checks.
- Splits: train 3,500, val 500, test 500 and calib 500; test CVs appear nowhere else and train/val/calib are CV-disjoint.
- Jobs come from public Greenhouse, Lever, Ashby and Workday pages, at most 10 per company; AGPL-3.0-or-later.
Quickstart
python
from datasets import load_dataset
root = "hf://datasets/gw0/jobfit-jevbench"
pairs = load_dataset("json", data_files=f"{root}/labels/pairs.jsonl", split="train").to_pandas()
test = load_dataset("json", data_files=f"{root}/splits/test.jsonl", split="train").to_pandas()
test = test.merge(pairs, on=["cv", "job"])Watch out
Card reviewed 2026-10-17; labels come from LLM judges rather than human recruiters, the card limits the dataset to non-commercial ML research and demo use, and job-post text keeps its publishers' rights. Only 25 distinct test CVs make per-CV statistics noisy, and *_likely_llm_generated is a model estimate, not ground truth.
Questions about this entry
Short answers drawn from the linked project or page.
A CV/job fit-scoring benchmark: 250 synthetic CVs, 404 job postings and 5,000 pairs, each judged on 17 typed questions with a score and a confidence. It trains and evaluates jobfit-model, a Jev-like typed-decision model.
The card describes jobfit-model as Jev-like: a state plus typed questions go in, and one typed answer per question id comes out, so labels are per-question scores with confidences instead of generated text.
Seventeen: 13 pairwise questions such as skills_match and overall_fit, two CV-level questions including cv_clarity_structure_quality, and two job-level questions covering clarity and likely LLM generation.
By LLM judges, one call per document and per pair, with claude-sonnet-5 as the primary rater and claude-opus-5-5 as a second rater on a subset; the card warns these are judge labels, not human recruiter judgments.
test is fully held out with its CVs appearing nowhere else, and train, val and calib are CV-disjoint from each other, though jobs are shared across all four splits; the card notes test holds only 25 distinct CVs.
AGPL-3.0-or-later with copyright of gw0, and the card says the dataset is for non-commercial ML research and demo use; job posts remain under their publishers' rights and would need per-source review before commercial use.
The CVs are synthetic and stylistically uniform, job postings come from a few tech-oriented companies, labels carry judge bias, and 25 test CVs make statistics noisy, so per-CV conclusions need bootstrapping.
More Jev resources like this
A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPythonEvaluation artifacts for hotchpotch's Bekko System One v0 models on Decision Index 0.2.1: local strict, no-truncation runs scoring 7.37 (17M), 12.83 (68M) and 21.87 (400M), with predictions and environment hashes.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
English, Turkish and German decision-training rows from the JevAlt project: 26,043 situations with typed questions and probability targets, used to train Deem-4B, Karar-4B and Wähler-4B; Karar-4B reports 96.8% on Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
A study posing speech-neuroprosthesis rescoring as a single typed Jev decision. On 978 held-out sentences from a participant with ALS, Jev reaches 7.5% word error versus 7.8% for OPT-6.7b and Qwen2.5-7B, at USD 0.07 per thousand sentences with no GPU needed.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.