Docs
CommunityJev Benchmarks & Evaluations
Jev Evaluation Traces
Raw traces from retired, versioned native ToolEmu and AgentDojo runs, including failures and partial trajectories, preserved as original bytes with SHA256 manifests and upload revisions.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- Ayushnangia
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- AgentDojo runs for clef, gemma4-12b-it, jev-omni and kev-27b, each with native-report.json, predictions.jsonl, smoke.exit and status.json.
- Retired native-v1 ToolEmu and completion manifests are kept under archive-manifests/.
- Files retain their original bytes; archive manifests record SHA256, sizes, source paths and upload revisions.
- Local trace copies are removed only after remote byte verification, and active files are never archived or deleted.
Watch out
The card declares no licence and the repository metadata carries only the region:us tag. These are retired runs that include failures and partial trajectories, not a completed 23-model evaluation, and the card states AgentDojo environments are synthetic benchmark fixtures with unvalidated ToolEmu judge scores. Gated classification source text is excluded from the archive.
Questions about this entry
Short answers drawn from the linked project or page.
Full raw traces from retired, versioned native ToolEmu and AgentDojo runs, including failures and partial trajectories, published so the exact artifacts each run produced stay reviewable.
Under aurora/agentdojo-results-before-storage-link, with folders for clef, gemma4-12b-it, jev-omni and kev-27b, each holding native-report.json, predictions.jsonl, smoke.exit and status.json.
No. The card states they are not a completed 23-model evaluation and not validated ToolEmu judge scores; they are archived raw runs that include failures and partial trajectories.
The card declares no licence, and the repository metadata carries only the region:us tag, so check the underlying AgentDojo and ToolEmu terms before redistributing anything from the archive.
Files keep their original bytes and archive-manifests/ records SHA256 hashes, sizes, source paths and upload revisions; local trace copies are deleted only after remote byte verification succeeds.
The card calls them synthetic benchmark fixtures, so the traces reflect model behavior on constructed agent scenarios rather than production traffic.
Gated classification source text is excluded from this native archive by design, per the card's notes on the release.
More Jev resources like this
A 30-option CWE weakness-class benchmark for Jev-class decision models: pick the class of a real NVD vulnerability description, with a 150-case dev split scored locally and 1,500 held-out test cases on the leaderboard.
Jev Benchmarks & EvaluationsDocs#community#evaluation#security
Community
Harmful/benign classification and answer-choice robustness for five type-safe models: 83,183 source requests from XSTest, OR-Bench and HarmBench in original, reversed and order-only variants.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
An independent Laya-versus-Jev comparison on one RTX 5090: phishing detection over 2,000 emails, the typed-decisions suite, latency scaling to 100 questions per pass and eleven probe assertions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPythonEvaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonThe benchmark behind JevAgentHarness's paper on decision models that share control with LLMs: next-tool decisions scored at isolated, open-loop and closed-loop levels, with 3,939 common-state questions and 153,621 answers from 39 models. The L1 question texts are third-party data and are not redistributed. CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#agents
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.