BennyLinntu/PACT-Pairwise-Anchored-Calibrated-Tuning-for-Single-Token-Typed-Decisions
User repository on GitHub
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions (arXiv 2609.35865)
PACT turns contrastive training pairs into four loss terms for single-token typed-decision models. On a frozen 324-item holdout it matches the published recipe's accuracy (84.6% versus 85.2%) while cutting answer flips under relabelling from 13.8% to 9.8%.
- Stars
- 3
- Language
- Python
- Format
- Article
- Published by
- Community
- Last verified
- Oct 2026
Highlights
- Adds four training terms with no new annotation: a difference-in-differences margin, permutation consistency, evidence necessity, and ordinal transport.
- Matches the published recipe in accuracy on a frozen 324-item holdout: 84.6% versus 85.2%, McNemar p >= 0.50 at every seed.
- Answer flips under relabelling fall to 9.8% from 13.8%, the lowest position bias of all runs.
- Ordinal error on rubric fields is MAE 0.232 versus 0.311 for the published recipe.
- Against a cross-entropy-only control it wins at two of three seeds, halves the seed spread, and lowers NLL by 26%.
Watch out
arXiv preprint dated September 26, 2026 (cs.CL, cs.AI, cs.CV, cs.GT) by Yida Lin. The evaluation is a single 324-item holdout with three seeds on one contrastively curated dataset; ablations show no single term raises raw accuracy, so the value is robustness and stability rather than accuracy.
Questions about this entry
Short answers drawn from the linked project or page.
It answers a schema question by reading the logits of a few one-letter answer codes at a single position, which is fast and returns a probability for every allowed answer.
Records curated as contrastive pairs: two contexts that differ in one edited fact which flips the answer, each carrying a machine-checked certificate that deleting the decisive sentence makes the fact unknown.
A difference-in-differences margin over each pair, a permutation-consistency term against answer-code position bias, an evidence-necessity term on certificate-verified ablated contexts, and an ordinal transport cost for rubric fields, plus a contextual temperature.
On a frozen 324-item holdout with three seeds it matches the published recipe, 84.6% versus 85.2%, with McNemar p >= 0.50 at every seed, so the gain is not in headline accuracy.
Robustness and stability: the lowest position bias of all runs (9.8% versus 13.8% answer flips under relabelling) and the lowest ordinal error on rubric fields (MAE 0.232 versus 0.311).
With the same optimiser and schedule, PACT is significantly more accurate at two of three seeds, halves the seed-to-seed spread, and lowers negative log-likelihood by 26%.
Seed-matched ablations and pre-specified falsification tests show no single term raises raw accuracy, locating PACT's value in robustness and stability rather than headline accuracy.
Related terms
Glossary definitions related to this entry.
More Jev resources like this
4GitHub stars
An analysis of ordinal-scale bias in JEV 1.13 and three open KEV models: across 36 ordinal datasets, final decisions use only 67-76% of the effective gold support versus 87-102% on four nominal tasks. BA-LoRA post-training raises utilization from about 47% to 86%.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Glax147/jev_ordinal_scale_biaPython
Glax147/jev_ordinal_scale_biaPythonA theoretical study of recalibrating many related yes/no questions that each have few labels, motivated by decision models such as Jev. It derives minimax compound-risk rates and shows pooled isotonic fitting with one per-question correction achieves them, while tuning questions separately loses a factor.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A training-free framework with two implementations that turn emotion classification into Jev judgments: one decomposes tasks into atomic judgments, the other aggregates complementary judgment paths. Across eight datasets standard Jev averages 62.93% macro-F1 versus 67.28% for the strongest LLM baseline.
Jev Practices & PatternsArticle#community#research#arxiv
Community
A study of option-order sensitivity in System One models and a training-free fix called Runoff. It scores each option alone to keep a few candidates, then compares them with each candidate in every position, removing order dependence in two forward passes.
Jev Practices & PatternsArticle#community#research#arxiv
Community
Shows that open instruction-tuned models already read decisions from answer-letter logits without training, and that LoRA fine-tuning with a decision head lifts them past Jev: Gemma 4 31B goes from 79.6 to 84.4 on the new 2,352-question OpenDecisions set and scores 81.5 on JevBench v1.7.
Jev Models & ReimplementationsArticle#community#research#arxiv
Community
Documents a failure mode in letter-bound remote sensing classification: 97.71% OA on UCM in vocabulary but 7.35%–96.40% cross-dataset accuracy and inverted confidence (AUROC 0.38). RSAJEV replaces the letter readout with training-free prototype retrieval, reaching 89.43% on the UCM cross-dataset gate.
Jev Practices & PatternsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
Training recipes, looped architectures, and language ports: what the open side of the typed-decision class shipped this month, and how to pick by runtime and calibration instead of the leaderboard.
Jev Models & Reimplementations#open-models#training#local
Read article
More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.
Jev Practices & Patterns#research#evaluation#calibration
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.