AbductionBench

AbductionBench is a unified framework whose taxonomy characterizes abductive tasks, the inference of plausible explanations for observed evidence. It harmonizes heterogeneous task formulations and scoring procedures so that claims about a model being good at abduction can be evaluated across more than one narrow aspect of the capability.

The framework is instantiated on 27 datasets spanning ten domains, and eight LLMs from four model families are evaluated on single-choice selection tasks.

Jev matches or exceeds small LLMs when selecting among explanations over compact evidence, but falls substantially behind on longer cases. The paper reads this as candidate discrimination and extended evidence integration placing different demands on a system.

It separates final-answer evaluation from analysis of the observable reasoning traces and interaction histories produced alongside those answers, characterizing evidence use, hypothesis comparison, uncertainty, contradiction handling, and answer commitment.

In interactive tasks the proportion of actions judged relevant declines at later steps, and a relevant first question is associated with a correct final answer.

In the passive task, models rarely revise their initial hypothesis as new evidence arrives.

The paper treats written traces as observable model outputs rather than faithful representations of internal computation, so trace analysis describes what models produce, not their underlying computation.