Every open decision checkpoint comes with the same warning in smaller type: the base weights are a starting point, not a decision engine. Laya's own card is the bluntest about it — the base English checkpoint scores 0.362 on the typed-decisions benchmark, below the 0.461 majority-class floor, and the 0.766 headline belongs to a checkpoint fine-tuned on that benchmark's training split. The gap between those two numbers is the work.
Zero-shot is not the point
This is a deliberate design choice in the encoder-first family. Laya, Julia 1, and OpenDecider Nano are fast bases to specialise: they read a state and score your options in one pass, and their accuracy on your workflow comes from what you train them on. Julia's card is equally candid — its 73.15% on typed decisions is real, and its Banking77 pilot still trails the supplied reference at 64 against 87.
The fine-tuning loop
The typed-decisions benchmark doubles as a training set: 400 cases across invoice processing, security incidents, customer service, and agent-trace observability, with 300 train and 100 test cases per workflow. Gold labels are the mean of three teacher samples, which is why the benchmark's ceiling is teacher self-agreement (about 0.735) rather than perfection.
Laya's fine-tuning notebook runs the whole loop on Kaggle's free 2xT4 GPUs in roughly four to five hours: build the dataset, train, fit calibration temperatures, evaluate, and push the checkpoint back to the Hub. If you need more supervision than 6,000 decisions, tasksource's recast holds 2.5 million typed decisions from 670 sources with per-row licenses, and the procedural set adds 572,000 rule-computed problems whose labels are noise-free by construction.
AutoTrust's corpus shows the shape at scale: 741,000 rows with full target distributions, split into train, validation, calibration, test, ood, and a 30,000-row held-out slice.
Calibration is a second fit
Fine-tuning moves accuracy; temperature fitting moves honesty, and they are separate steps. Laya reports mean ECE falling from 0.466 to 0.081 on the English checkpoint after refitting one temperature per question type and option count — temperature scaling learned on held-out data, not a property the base weights shipped with.
The pattern repeats across the ecosystem. Mapika's readout uses a temperature that varies with option count, because Gemma-4 instruct models are overconfident with few options and underconfident with many. GEV-26B-Decide ships two calibration files: one for the leaderboard, and a "gold" variant for gating real actions. And the PT-BR benchmark shows why the fit belongs to your data: its tuned 4B model reaches 72.9% mean where TF-IDF scores 64.0%.
When not to fine-tune
Tuning is not free. Lev's card documents a 200,000-example recipe and notes that fine-tuning reduced summary consistency to 0.271 where the untuned backbone scored 0.826. Distilled students inherit their teacher's failure modes along with its calibration — counting, dates, arithmetic, adversarial inputs — because the training target is the teacher, not the truth.
So measure the tasks you did not train on before swapping anything out of production. Accuracy on your new workflow, ECE on a held-out split, and a threshold you can defend are three different numbers, and only the last one gates an action. For how the exports themselves work, see exporting decision models; for where these checkpoints fit next to the hosted API, see Open weights on Hugging Face.