Open decision models are easy to download and hard to adopt. The weights are one artifact; the license, the training data, the evaluation claims, and the maintenance commitments are separate artifacts, and they do not always travel together. A model card that says Apache-2.0 may be describing the adapter, not the backbone. A benchmark that says “beats Jev” may be measuring a different task on a different version.
This is the supply-chain checklist for an open decision model: what to verify before you put it behind a production gate.
Read the license at the right layer
Most open decision models are built on a base model plus an adapter or head. The base model's license and the adapter's license can differ. Soro-Kev-4B is a useful example: Apache-2.0 covers the adapter and head, while the Qwen3-4B backbone keeps its own license. That distinction matters if you plan to redistribute the merged model or run it as a service.
Laya is Apache-2.0, developed by Convai Innovations, and is not affiliated with TypeSafe. Kev-0.8B is a LoRA plus pointer head on Qwen3.5-0.8B-Base; the card documents the adapter, but the base model's terms still apply. openjev is MIT licensed on Qwen3.5, with the same layered question.
The practical rule: identify every layer (base model, adapter, tokenizer, head, training data) and read the license for each. If the card only says “Apache-2.0,” check whether that is the whole package or the adapter alone.
Check what the card actually measured
A model card is a claim, not a certificate. The Typed Decision Models evidence audit reviewed 28 papers posted in the days after Jev's release and found that the typed readout had not shown an independent accuracy advantage over comparable label-probability readouts; the clearest gains were latency and cost. That is a useful prior when you read a new card.
The AutoTrust JEV-35B card is a good example of detailed self-reporting: Decision Index 0.3 public 59.73, ERQA 51.3% in one pass, 66.5 GiB of bf16 weights, 204 ms median latency on a B200. The caveat is that the numbers are AutoTrust's own runs with the Decision Index kit, not a board entry, and the ERQA 400-question set carries roughly a ±5-point interval.
Read the evaluation section for three things: who ran it, which version of the model and benchmark, and whether the baseline is matched. A number without those three is a pointer, not evidence.
Verify the context limit
Cards often advertise a larger context than the model was trained or validated on. Soro-Kev-4B advertises 32,768-token states, but the caveat says training states were capped at 4,096 tokens and 8,192 were validated. Kev-0.8B serves up to 65,536 tokens but documents 8,192 as validated.
That gap is not necessarily a dealbreaker. It is a reason to test at the length you actually plan to use. A model that works at 4,096 tokens can degrade quietly at 16,000, and the card may not say so.
Check the calibration artifacts
A probability is only useful if you know how it was fitted. Look for a temperature, a conformal quantile, a calibration split, and a caveat about out-of-distribution behavior. Train-Jev 27B LoRA reports ECE 0.053 against 0.097 for the official model, with the caveat that calibration holds only in distribution and the results come from a single seed. Soro-Kev-4B reports a learned temperature of 1.62 and a split-conformal quantile of 0.8066 at 90% coverage.
Those are the numbers that tell you whether a gate can trust the probability. If the card has no calibration section, assume raw logits and plan to fit your own temperature on a held-out split.
Check the maintenance surface
An open model is a dependency. Look at the release cadence, the issue tracker, the license file, and whether the card links to code. Laya publishes checkpoints and a router; Kev-0.8B documents serving and hardware; openjev publishes a model card with candid security weaknesses.
A card with no code, no license file, and no issue tracker is a demo, not a dependency. That does not make it useless; it makes it a research artifact.
Run your own audit
The open-model audit is a small version of the vendor audit:
| Check | What to do |
|---|---|
| License | Read every layer, including the base model and data |
| Provenance | Check the base model, the adapter, and the training data |
| Claims | Reproduce one headline number on your own sample |
| Context | Test at the length you plan to use |
| Calibration | Check for a temperature or conformal artifact |
| Maintenance | Look at releases, issues, and code links |
| Security | Test the model on adversarial inputs you care about |
The awesome-typesafe list and the open decision-model roundup are good discovery layers. The choosing an open decision model article covers the selection process. The audit is the step after selection.
A short checklist
- Read the license for the base model, adapter, head, and data separately.
- Check who ran each benchmark and on which version.
- Test the context length you actually need.
- Look for calibration artifacts and fit your own if they are missing.
- Verify code, releases, and issue activity.
- Reproduce one headline number before trusting the rest.
- Test adversarial inputs, not only clean examples.
Open weights are a supply chain. Treat the card as the manifest and verify each part before you depend on it.
