Swap, Replicate, Read the Mass

The paper provides a multi-model audit of System One decision models on established bias benchmarks: BBQ across all eleven categories, CrowS-Pairs, and SilverSpoon, all adapted to the typed interface. It reports widespread stereotype alignment, order-dependent answers, and confidence that tracks instability rather than bias.

The audit covers twelve models plus a no-thinking ablation and 1,638,000 decisions. Ten configurations were repeated under extra seeds and batch 16.

Every order-sensitive item runs twice with the options swapped, and native typed models also swap labels. The authors argue audits must permute options and reverse questions by default, and that order-invariant scoring should become a procurement requirement.

In BBQ, 97 of 143 model-by-category scores are significant and 94 are aligned. The commercial reference abstains on 99.5% of ambiguous items, which the authors say hides aligned probability mass.

Option swaps flipped between 0.0% and 72.6% of choices. Typed scoring stayed below 10% except for Laya at 22-33%, while untuned letter prompting reached 73%. Boolean answers could be decided by position, at 1.00 versus 0.005.

SilverSpoon judgments shifted toward lower-SES labels in five of eight decision models. The paper attributes this either to the 9B backbone or to tuning.

The paper recommends that audits permute options and reverse questions by default, and that order-invariant scoring become a procurement requirement. It also notes that CrowS-Pairs preference spans 0.496 to 0.816 across the audited configurations.