Jev Robustness

A study of option-order sensitivity in System One models and a training-free fix called Runoff. It scores each option alone to keep a few candidates, then compares them with each candidate in every position, removing order dependence in two forward passes.

A pre-registered study of whether jev-1.13.0 satisfies independence of irrelevant alternatives, using 299,463 calls on intent and topic classification. Every added option shifts the log-odds between the gold and strongest wrong label by 0.20–0.52 nats RMS, 1.2 to 3 times run-to-run variation.

PACT turns contrastive training pairs into four loss terms for single-token typed-decision models. On a frozen 324-item holdout it matches the published recipe's accuracy (84.6% versus 85.2%) while cutting answer flips under relabelling from 13.8% to 9.8%.

An evaluation of Jev and two open-weight typed decision models when option names and definitions conflict. With yes/no names, decision-flip rates are up to 70.4 percentage points higher than the 0/1 control, and mean AUC drops from 93.8% to a below-chance 23.2%.

JevOut shows short background context additions can flip correct decisions in four systems including Jev across seven datasets. Within 64 accepted evaluations per decision, additions redirect 61.4%-73.2% of initially correct decisions toward a wrong option fixed in advance.

A comparative study of system-one decision models across 15 medical datasets and 8,742 instances, measuring accuracy, decision stability, and abstention. Jev rivals leading models on text-driven tasks and is weaker on numerically driven ones, while most models still fail to recognize missing answers.

An evaluation of Jev and nine language models on ContractNLI with repeated requests that vary visible hypotheses, requested answers, and output order. Corrections and regressions can cancel so accuracy stays unchanged while decisions change; Jev is cheapest and fastest for the joint baseline.

An evaluation of Jev on training-free human activity recognition using 1,800 class-balanced accelerometer windows from WISDM, UCI341, and PAMAP2. Its strongest representation reaches macro-F1 of 0.038, 0.118, and 0.089, versus 0.686 to 0.907 for the supervised models.