Most decision evals are underpowered. They compare two systems on a few hundred cases, run each once, and report a ranking with no interval. The ranking may be real, but the design often cannot distinguish a stable difference from run-to-run variation, case difficulty, or the choice of a threshold. The fix is ordinary statistics: define the unit, pair the cases, report intervals, and treat a threshold as a parameter to validate rather than a constant to import.
The unit of analysis
The first decision is what counts as one observation. A workflow eval with 150 invoice cases and 204 customer-service cases has 150 and 204 independent episodes, not one number per question. The WorkflowEvals code makes the shape explicit: invoice processing has 150 cases, customer service 204, agent traces 111, and security incidents 240. The published evals score each case against consensus labels from GPT-6 Astra and Claude Fable 5.1, and report 67.8% for Jev at $0.0004 and 0.4 s per case.
If each case carries eight questions, the question count is eight times larger but the effective sample size is closer to the case count, because questions within a case share evidence and errors. Report the case count and the clustering, not just the total judgments.
| Workflow | Cases | What one case is |
|---|---|---|
| Invoice processing | 150 | One invoice episode |
| Customer service | 204 | One support episode |
| Agent traces | 111 | One recorded trajectory |
| Security incidents | 240 | One incident |
With 150 cases, a 5-point accuracy difference is 7.5 cases. That arithmetic does not tell you whether the difference is significant, but it shows how little data sits behind a headline gap.
Why single runs mislead
The IIA study is a useful noise reference because it ran 299,463 calls. Every added option shifted the log-odds between the gold label and its strongest wrong label by 0.20–0.52 nats RMS, about 1.2 to 3 times run-to-run variation. The effect is real, but it is only a few times the noise. A single shuffled run could easily show a shift in either direction and be read as a clean result.
TypedBench handles this by reporting accuracy as a median and range across paraphrases and by judging calibration against the finite-sample noise floor of a matched, perfectly calibrated predictor. That is the right comparison: not "is the error zero" but "is the error larger than what a finite sample would produce." It also finds the hosted model systematically underconfident, which changes where a threshold should sit.
JEVal evaluates 25 model configurations on 11,257 bilingual instances from 36 datasets, which is enough data to compare broad families. Its caveat is that the abstract reports benchmark results without run-to-run detail or third-party replication. Breadth does not replace repeated runs.
Intervals, not point ranks
A ranking is a point estimate. What you need is the interval around the difference, preferably from a paired design where both systems see the same cases in a randomized order. Pairing removes case difficulty from the comparison; randomization removes order effects. If the interval includes zero, the rank is not established by that experiment.
The 37-dataset evaluation is a good example of reporting the comparison rather than only the win count: Jev beat Qwen3.8-27B on 27 of 37 datasets, with none of Qwen's nine leads outside the bootstrap intervals. The number 27 is less informative than the interval statement. The paper also uses one frozen template per dataset, which controls a source of variance but limits what it says about paraphrase sensitivity.
For a single workflow, report the paired difference, the interval, and the cost and latency intervals. A 67.8% versus 74.1% gap on 150–240 cases may or may not be stable; the report should let the reader see that. If you compare many workflows or many datasets, say how you handled multiple comparisons rather than picking the significant rows.
Pooling questions with few labels
Recalibrating Many Questions with Few Labels is the theoretical version of a common practical problem: many related yes/no questions, each with only a handful of labels. The paper derives a minimax compound risk for maps near one shape and shows that a pooled isotonic fit with a single per-question correction attains it without knowing the shape parameters. Tuning each question on its own labels loses a factor of order the smaller of n and log(m squared over n) when the maps differ by constants.
The identification result is separate. Once a question's data law is known, identification regret is at most 2 to the minus n; the worst case is at least beta-star to the minus n over 72 times the square root of n, with beta-star about 3.583. The practical reading is that pooling is not just a convenience. With few labels per question, sharing shape across related questions is what makes calibration estimable, while per-question corrections absorb the differences.
The caveats matter: the results are theoretical, stated up to universal constants for non-atomic output laws, and they do not establish how the rates behave on a real hosted model with discrete probability outputs. Treat the paper as a reason to pool and hold out, not as a recipe with a guaranteed constant.
Threshold transfer is a power problem
Rejection-policy transfer shows what happens when a threshold fitted on one task is moved to another. A Jev policy calibrated on DBpedia rejected 69.3% of covered Emotion test inputs; an Emotion policy lost detection entirely. With ten intent candidates, Laya detected 99.0% of out-of-scope queries but rejected 48.8% of covered queries. The comparison uses equal calibration budgets, so the failure is not a budget artifact.
A threshold is a parameter estimated from data. Transferring it without target labels is a power problem because you have no estimate of the target operating point. The confidence guide says the same thing in product terms: threshold values depend on your domain and the model's performance on your use case, and the docs advise starting conservative and tuning against your own data. The guide's example routes below 0.5 to a human and requires above 0.9 to auto-approve, but those are illustrative.
A power checklist
- Define the smallest difference you would act on before running the eval.
- Pair the systems on identical cases and randomize the order.
- Count independent units, not total judgments; report clustering.
- Run multiple seeds and paraphrases, and report the spread.
- Report paired intervals for accuracy, cost and latency.
- Pre-register the threshold and hold out target labels for it.
- Adjust for multiple comparisons, or label the analysis exploratory.
- Treat a transferred threshold as a hypothesis, not a setting.
The operational version of this is in reproducibility, model, question and policy pinning. If the threshold feeds a gate, confidence gates in production covers the monitoring side, and measuring Jev covers the field-report numbers. The confidence, threshold and repeatability entries define the terms.
