ASSAY-001

Both, and the split is the finding. Chosen-option probabilities were calibrated on CLINC150 at ECE 0.0204 and were not on Banking77, where ECE was 0.0936 and the model was systematically overconfident. Read that as task-dependence rather than a contradiction: the report's own closing line is that a result applies to the artifacts and criteria examined, not to Jev on any other task, corpus or day.

Yes, with one disclosed wrinkle. PROTOCOL.md was frozen on 2026-09-17 before any query, with a FREEZE.sha256, and the Banking77 and CLINC150 corpora were pinned to specific commits and hashed before the first call. AMENDMENT-1 landed before the run; Amendment 2 landed after and is disclosed inside REPORT.md. Both rule sets' scores are kept in scores/ under jev-frozen-* and jev-* prefixes, so you can see what each rule produced.

Yes for the scoring. Run harness/controls.py first, which makes the scorer go red and green on positive controls, then harness/score.py against both corpora at --sum-tol 0.02. A blind re-score written from a spec file on a different base model matched every field of the original. Only re-running the model needs a key at ~/.config/typesafe/api_key, and that produces a new run rather than the recorded one, so verify the log you already have.

It is a real count from sealed artifacts: one Choice question per item with null option descriptions, every request and response logged verbatim under responses.sha256, and the 3-item smoke connectivity runs excluded from scoring. But type safety is only half the claim, and it is the easy half. The same 8,576 responses are where Banking77 calibration fails, which is the number that should gate a decision.

PROTOCOL.md with its FREEZE.sha256 plus AMENDMENT-1 and Amendment 2; corpus/ with CORPUS.sha256 and PINS.txt; harness/run.py, harness/score.py and harness/controls.py; raw/run/<corpus>/responses.jsonl sealed by responses.sha256 with raw/smoke/ kept apart; scores/ holding both rule sets plus controls; rescore-tifa/ for the blind independent re-score; and REPORT.md with GATE.md. The full write-up lives at donttrustme.ai/assay-001.html.

Both are pinned by commit rather than by name. Banking77 is GitHub PolyAI-LDN/task-specific-datasets at commit 57ec275d, and CLINC150 is the plus configuration on Hugging Face clinc/clinc_oos at commit 155b9c71. Both were hashed into CORPUS.sha256 before the first API call, which is what makes the pre-registration claim checkable rather than merely asserted.

One Choice question per item, with null option descriptions so no label text can leak the answer, and run.py logging every request and response verbatim. Scoring is deterministic in score.py, gated on positive controls in controls.py, and the three-item smoke connectivity runs live in raw/smoke/ so they are excluded from every reported number. ASSAY is a JourdanLabs verification practice.