Docs
CommunityJev Benchmarks & Evaluations
AgentDecisionBench
The benchmark behind JevAgentHarness's paper on decision models that share control with LLMs: next-tool decisions scored at isolated, open-loop and closed-loop levels, with 3,939 common-state questions and 153,621 answers from 39 models. The L1 question texts are third-party data and are not redistributed. CC BY 4.0.
- Format
- Docs
- Reading time
- 3 min
- Published by
- Community
- Author
- JevAgentHarness
- Last verified
- Oct 2026
- Capabilities
- Open source
Highlights
- Three levels: L1 isolated (3,991 BFCL v4 and When2Call items), L2 open-loop over recorded states, L3 closed-loop run by the code.
- The default common_states config holds 3,939 distinct L2 questions, with an answer key of 5,314 trajectory occurrences.
- common_states_answers carries 153,621 rows (39 models); isolated_answers holds 167,622 rows from 42 models.
- DecisionGym generates customer-service tasks with 60 near-miss distractor tools; Shift renames tools and adds 120 distractors.
- L1 question texts are third-party data and not copied; answers include confidence, cost and latency fields.
Watch out
Card reviewed 2026-10-17; L1 question texts are withheld because they are third-party data, closed-loop L3 requires running the research code with an agent LLM, and the leaderboard config only copies the paper's own generated tables. The card's answer-field list also ends without enumerating every column of that schema.
Questions about this entry
Short answers drawn from the linked project or page.
The benchmark of the JevAgentHarness paper on decision models that share control with LLMs: a small model answers a multiple-choice question about an agent's next step with a probability per option, and a harness can hand that choice to the model when it is confident and let the LLM decide otherwise.
L1 (isolated) scores standalone questions such as BFCL v4 tool choice and arguments and When2Call, with no agent around them. L2 (open-loop) scores answers at states recorded from real runs against an answer key. L3 (closed-loop) lets choices drive the running agent, scored by task success.
DecisionGym generates customer-service tasks - find the customer, look up orders, cancel, ship, refund - with 60 near-miss distractor tools and noisy tool output, deriving an answer key of acceptable tools per state. DecisionGym-Shift renames tools, adds 120 distractors and new task kinds.
common_states has 3,939 L2 questions and its answer key 5,314 trajectory occurrences; shift has 1,093 questions with nosurf and nodist re-renderings; common_states_answers holds 153,621 rows from 39 models, isolated_answers 167,622 rows from 42 models, and leaderboard copies 169 paper numbers.
The card maps the paper's names to run ids: Jev is jev (typesafe/jev-1.13), the teacher of the training data. AgentDecision-4B is e12, its earlier checkpoint is rl (JevAgentRL-4B), and AgentDecision-0.8B is e28, a StartLux-Decision-0.8B plus recipe.
L1 question texts: they are third-party data and are not copied into the repository, though each model's answers and correctness on 3,991 items are. L3 runs require the research code and an agent LLM, and the leaderboard config merely copies the paper's generated tables.
CC BY 4.0 for the dataset, with the research code in the linked GitHub repository. Because L1 question texts are withheld and L3 is code-level, the repository alone supports studying L2 decision records, answer keys and model answer distributions.
More Jev resources like this
Twelve million canonical agent-decision records built from public agent trajectories, canonicalizing state plus candidate decisions into target, outcome and eligibility supervision under CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Evaluation scripts, MCP tools, and a skill router built to measure where jev-1.13.0 fits in agent harnesses, with raw results under eval/.
Jev Benchmarks & Evaluations#community#evaluation#agents
Community
Aitejiu/jev-harness-labPython
Aitejiu/jev-harness-labPythonA local Java benchmark that plays 10,000 deterministic transactions through two fraud pipelines: every one to Qwen via Ollama, or through System One/Laya first, where a low-risk answer at confidence 0.70 or higher stops before the chat model.
Jev Benchmarks & Evaluations#community#github#java
Community
AliveMonster/JEVJava
AliveMonster/JEVJavaEnglish, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Arabic intent-routing benchmarks split across MSA, Emirati, Saudi and code-switched slices: 4,974 held-out queries over 60 routes plus a 20-question unseen-domain RAG test, CC BY 4.0.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
A benchmark of natural-language requests to edge services paired with typed intent contracts: 33 conditions of 300 test cases each, from a UTS study on Jev decision models for edge orchestration.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.