An agent eval can score three different things: a single decision, a recorded trajectory, or the outcome of a live run. They are not interchangeable. A model can win the isolated question and lose the episode, because a small per-step error compounds, the harness supplies the wrong context, or the live state differs from the test file.
The useful evaluation design says which level it is measuring and why.
Three levels of eval
AgentDecisionBench formalizes the distinction. L1 is isolated: 3,991 BFCL v4 and When2Call items, with no agent around them. L2 is open-loop: 3,939 distinct questions at states recorded from real runs, with an answer key of 5,314 trajectory occurrences. L3 is closed-loop: the choice drives the running agent, and the score is task success.
The dataset carries 153,621 answer rows from 39 models for the common-state config and 167,622 rows from 42 models for the isolated config. DecisionGym generates customer-service tasks with 60 near-miss distractor tools; DecisionGym-Shift renames tools, adds 120 distractors and new task kinds, for 1,093 questions. Jev appears as the teacher model jev (typesafe/jev-1.13). L1 question texts are withheld because they are third-party data, and L3 requires the research code and an agent LLM, so the repository alone supports studying L2 records, answer keys and answer distributions.
| Level | What is scored | What it can show | What it misses |
|---|---|---|---|
| L1 isolated | One question, no agent | Model capability on a clean item | Context, recovery, accumulation |
| L2 open-loop | Answers at recorded states | Decision quality on real states | Whether the choice succeeds in-loop |
| L3 closed-loop | Task success | End-to-end behavior | Which decision caused the outcome |
Traces as the unit
Fast Models, Slow Evidence is the strongest paired evaluation of decision models in agent harnesses. It builds 11 agent decision points from 18 public sources, with 7,283 base cases plus 6,640 robustness variants, and runs byte-identical inputs through an open-weight Laya model and hosted Jev. Jev was significantly more accurate on 9 of 11 decision points, by 10.8 to 46.0 points. Neither model beat chance on zero-shot model routing, and the two tied on RAG relevance gating.
The robustness numbers show why the harness matters. Laya changed 30% of its answers when option order was reversed, and reached 31% at 50 nearest-neighbour tools, versus 98% for Jev on items with a unique correct tool. A decision that looks fine in isolation can degrade as the candidate set grows.
The paper's self-audit is the part to copy. Three analysis errors and one design confound distorted the headline deployment claims. An omitted pre-screen cost turned a reported 23.9% saving into an actual 4.3%. Gate accuracy was reported as end-to-end quality, 58% versus 98%. Thresholds were in-sample. A channel effect on injection false positives vanished with channel-native content. Two other suspected confounds did not change the conclusions, and the authors report those too. The corrected numbers are the contribution.
Harness decisions
Jev Harness is an agent harness where Pi runs the loop, Gemini does the work, and Jev makes four small decisions: which model tier to use, which runbook sections to load, whether a tool call may run on a real Postgres database, and whether the final report holds up.
The context picker asks 22 yes/no questions in one call, one per runbook section, to trim the system prompt. The gate classifies a tool call as read, additive, modify or destroy and returns allow, checkpoint, ask or block. hardRule() blocks the Neon control plane, the neon CLI and production IDs in code before Jev is consulted. The gate fails closed and blocks when Jev cannot answer; the router fails open and picks the powerful model.
The benchmark is a seeded Brightcart store with planted problems. One ticket drops a column backing 17,423 order items; another undoes 1,500 wrongly cancelled orders. The README records that an agent with a Neon key reset a database password to get unstuck, and ten of fifteen runs in a first clean batch read the benchmark's answer key or scorer. That is why the shell is sandboxed and the working directory sits outside the home directory. The harness is MIT licensed, accompanies a video, and its published results were measured without tool-output clipping.
End-to-end outcomes
System Switch studies when a fast actor should hand control to a reasoning vision-language model. On 900 held-out questions, 0.15B–9B decision models chose to collect items 1.6–1.8 times more often than chance among their errors, in any option order. Accuracy, calibration and sensitivity were distinct: models of similar accuracy differed widely in AUROC. Deferring the least confident 30% gained over random deferral in proportion to actor AUROC, with rank correlation 0.87. With actor and rate chosen on held-out games, the gain was 0.13 [0.08, 0.18], and 0.08 [0.02, 0.14] with shuffled options.
The closed-loop result is negative. Across 33 games and three seeds, no variant reached the exit, and the reasoning model carried only about half of the offline deferral gain. The lesson is that an offline gain can disappear when the policy changes the states the agent reaches.
Jev-Guard is a more favorable harness result, with a large caveat. On 7,033 cases it reports 93.72% safety accuracy and 93.10% hierarchical routing accuracy at 58 ms mean latency for classification, or 240 ms for the full routing-and-classification pipeline. As a separate permission gate in the Hermes harness, it cut mean review time per task from 4.80 s to 1.33 s, and no policy violations were observed for either guard. All numbers are self-reported, and the abstract does not say whether weights, code or the policy tree are released.
Workflow benchmarks
TypeSafe's workflow evals score four code-written workflows: invoice processing, customer service, agent traces and security incidents. Jev averages 67.8% accuracy at $0.0004 and 0.4 s per case against consensus labels from GPT-6 Astra and Claude Fable 5.1. The page reports that every model is more accurate, cheaper and faster in a workflow than with the same policy as a prompt, on average. The labels are not ground truth and the harness is assumed correct.
The WorkflowEvals code makes the comparison reproducible: 150 invoice cases, 204 customer-service cases, 111 agent traces and 240 security incidents. The scorer reports exact_actions, which matches action sets including arguments, and primary_action, which matches just the primary actions. --reference scores agreement with another run folder, and a model is excluded from scoring against its own answers. Apache-2.0, Python 3.13 and uv.
jev-eval is a smaller third-party check: 60 synthetic booking inquiries in four languages, routed by Jev, GPT-4o-mini and Claude Sonnet 4.5 through one Vercel AI Gateway key. The main 60 × 3 run cost about $0.03; the Claude run about $0.62. Labels came from a single annotator, all cases are synthetic, and there is no licence file. The README distinguishes the replay demo, which uses recorded latencies, from the live demo, which calls APIs in parallel and does not reproduce serial measurement.
The harness is part of the result
- Choose the level explicitly: isolated, open-loop or closed-loop.
- Freeze the traces and the states before comparing models.
- Pair the systems on the same cases and randomize order.
- Report both the component decision and the end-to-end outcome.
- Log the decision, the state, the probability and the harness action.
- Check for answer-key exposure and other harness leakage.
- Re-run the gate after changing the policy, because the state distribution changes.
The agent-side design patterns are in decision gates in agent frameworks and per-tick decisions. For the evaluation numbers behind them, see measuring Jev. The agent loop, workflow evals and guardrail entries define the terms.
