Docs
CommunityJev Benchmarks & Evaluations
Jev Decisions v1
Twelve million canonical agent-decision records built from public agent trajectories, canonicalizing state plus candidate decisions into target, outcome and eligibility supervision under CC BY 4.0.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- samatv256
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- 11,978,080 usable canonical records: 6,214,185 choice-supervised, 9,369,747 value-supervised and 48,293 completion-supervised rows over 1,066,352 tasks.
- Group-aware 94/3/3 split with verified zero trajectory overlap: 11,269,408 train, 354,037 validation and 354,635 test records.
- CLEAN-50K is a sanitized 50,000-decision Choice-only subset: 30,000 Nemotron-SFT-Agentic-v2, 17,217 conversational tool-use and 2,783 function-calling rows.
- Every published CLEAN-50K branch passed the 8,192-token policy measured with the exact Qwen3-0.6B tokenizer.
- The original canonical Parquet artifact is 48,763,693,033 bytes (45.415 GiB).
Watch out
CC BY 4.0 over a derivative of NVIDIA public datasets, so source and component terms in SOURCE_LICENSES.md still apply and NVIDIA is not an endorser. CLEAN-50K is a sanitized derivative of the private selection rather than the exact training copy: 11,368 rows were redacted, 32 rows duplicate other rows' model-visible text, and older mini-Jev baselines were not trained on this corpus.
Questions about this entry
Short answers drawn from the linked project or page.
A canonical interface of state plus available candidate decisions mapped to a target, outcome or eligibility label, derived from public agent trajectory datasets and meant for action selectors, routers, completion policies and value models.
11,978,080 usable records over 1,066,352 unique trajectories: 6,214,185 choice-supervised, 9,369,747 value-supervised and 48,293 completion-supervised rows, with zero score-supervised rows.
A sanitized 50,000-decision Choice-only training subset taken from the public train partition at revision d063f00b, loaded with config general-clean-50k; it contains no Open-SWE records.
Four NVIDIA sources: Nemotron-SFT-Agentic-v2, Nemotron-RL-Agentic-Conversational-Tool-Use-Pivot-v1, Nemotron-RL-Agentic-Function-Calling-Pivot-v1 and Open-SWE-Traces v1.0 for the full corpus; CLEAN-50K draws on the first three.
A deterministic group-aware split keeps all rows of a trajectory or task in one partition, about 94/3/3, with verified zero overlap: train 11,269,408, validation 354,037 and test 354,635 records.
Lengths follow an 8,192-token branch policy checked with the exact Qwen3-0.6B tokenizer; the question, candidate definitions, user goal, current environment and latest observation are protected while earlier history may be shortened.
The dataset-level licence is CC BY 4.0, with attribution and applicable source terms in SOURCE_LICENSES.md; the card notes NVIDIA is not the publisher or endorser of the CLEAN-50K subset.
More Jev resources like this
The benchmark behind JevAgentHarness's paper on decision models that share control with LLMs: next-tool decisions scored at isolated, open-loop and closed-loop levels, with 3,939 common-state questions and 153,621 answers from 39 models. The L1 question texts are third-party data and are not redistributed. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonA reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonnisaralll's dataset for training and calibrating System-One pointer routers: 23,192 route, argument-repair and safety rows over six tool-declaration formats, plus zero-shot and human validation splits. Tasks are formatted as pointer queries over explicit option lists. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#tool-calling
Community
Run outputs behind the JevGPT paper's tables: 830 BrowseComp-Plus records at 90.48% accuracy and 105 NeuCLIRBench topics at 0.5390 nDCG@20, produced by a GPT-5.5 System Two paired with Jev-1.13.0 as System One. Per-item cost, latency and encrypted trajectory files are included. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
cnfeichen's reproducible Chinese reranking benchmark for Jev over hard negatives from C-MTEB/T2Reranking: a 4-level Jev Score rubric reaches 0.546 nDCG@5 on 100 queries against 0.413 for BM25 with jieba. Independent community research, not affiliated with TypeSafe. MIT.
Jev Benchmarks & EvaluationsDocs#community#evaluation#reranking
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.