A model card is a claim set, not an audit. It tells you what the publisher chose to measure, on which split, against which comparator, and with which caveats. Before trusting a checkpoint with a production decision, separate what was measured from what was asserted, find the lineage, and check whether anyone outside the publisher has rerun the headline.
Read the claim, then the comparator
Start with the exact sentence. Soro-Kev-4B reports 89.00% zero-shot tool routing, 94.50% on external human holdouts, and 75.76% calibrated abstention on BFCL irrelevance slices. Those are self-reported with no third-party reproduction. The same card says training states were capped at 4,096 tokens and validated at 8,192, even though the server advertises 32,768. The head is Apache-2.0; the Qwen3-4B backbone keeps its own licence.
Train-Jev 27B is subtler because the headline sounds like a tie. It reports 0.851 Kev transfer-v4 accuracy against 0.852 for the official JEV-27B-VL, which is not a win. The interesting claim is calibration: ECE 0.053 against 0.097, NLL 0.411 against 0.460, and Decision Index 54.6 against 54.1. The card is explicit that the table is self-reported, calibration holds only in distribution, only labels A–P are trained, and results come from a single seed. A tie plus better calibration is a result; a tie presented as a win is not.
AutoTrust JEV-35B shows how comparator choice changes the story. It reports Decision Index 0.3 public 59.73 against JEV-27B-VL's 53.64, both scored with AutoTrust's own kit, not a board entry. ERQA is 51.3% in one pass with zero robot training, level with PhysBrain 1.5's 52.8, but ERQA has 400 questions and the card says the 95% interval is about ±5 points. The repo ships a LICENSE file whose terms the card does not name, and the model is explicitly not TypeSafe's Jev.
OpenJev is the useful counterexample because the author publishes failures beside wins. It scores 0.814 on JevBench v1.2 public items against Jev 1.13's 0.866, and 0.622 on the hard tier against 0.730. Option order changes zero labels over 231 items, with a largest probability shift of 1.8e-7. Then the card reports 0.600 accuracy as a shell-command guard, catching 37% of deny-worthy commands, and one adversarial line dropping accuracy from 0.833 to 0.467. That is a candid card.
Checkpoint lineage and components
A checkpoint is a bundle, not a single file. Train-Jev ships the LoRA adapter, head.pt for the 24-slot head, and calibration.json; the backbone downloads separately, and plain PEFT or Transformers loading applies the adapter but skips the head and temperatures. If your loader does not know about the head, you have not loaded the model.
Soro ships a rank-16 LoRA and a pointer head over Qwen3-4B. JEV-35B is a full bf16 checkpoint at 66.5 GiB, with a separate 72 GB download. open-jev is a sequence-classification model on Qwen3.5, loaded with AutoModelForSequenceClassification or the project's OpenJev.decide helper. The Decision Index results are not a checkpoint at all: they are engine outputs for three models, with no benchmark inputs, so you need the index kit to rebuild the suite.
Lineage also means the base model, revision, and tokenizer. kev-4b-mlx-fp16 is a good template even without benchmarks: it pins kev-4b at revision 6cfce5c2fa4b4bd64026336ab649c5ca78857d52, records every file's SHA-256 and size in package.json, and states the package digest. If a card gives you a moving main branch and no hash, you cannot tell which weights produced the number.
Are the benchmark numbers reproducible?
The early evidence audit reviewed 28 papers posted in the first nine days after Jev's release and derived a 14-item checklist. Its uncomfortable finding: in that early literature, the typed readout had not shown an independent accuracy advantage over comparable label-probability readouts. Jev's clearest gains were latency and cost. The burden of proof is on the card.
The 37-dataset benchmark is the strongest outside evaluation in the catalog. It runs jev-1.13.0 zero-shot on 37 datasets, 346,009 requests for under USD 10, and reaches 95–99% on IMDB, SST-2, HellaSwag, and ARC, plus 86.7% on Belebele across 122 languages. It beats Qwen3.8-27B on 27 of 37 datasets and Gemma-4-E4B on all 37. But all three models degrade on low-resource languages, noisy labels, and rubric judgments. Choice probabilities are well calibrated; binary probabilities rank well but need tuned thresholds, with micro-F1 on UNFAIR-ToS moving from 0.50 to 0.75 after tuning.
The Decision Index results are a complete run, not a card summary. Three models cover 150,317 of 150,317 scoreable requests with zero errors, scoring 43.14, 53.30, and 58.05. Published artifacts include scores.json, index.json, environment records, engine source with SHA-256, and a training-overlap check reporting no whole suite state, instruction, or option in training. But only engine outputs are published, without benchmark inputs, and the batched phase reports wall time divided by 64 rather than per-request latency.
Calibration and confidence claims
A card's calibration number is only meaningful with its fitting procedure. The confidence documentation makes the distinction concrete: confidence is a derived 0–1 statistic, Noul answers carry only a probability of yes, and the full distribution comes back. The documented example routes below 0.5 to a human and requires above 0.9 to auto-approve a transfer, but those values depend on your domain.
Soro publishes a learned temperature of 1.62 and a split-conformal quantile of 0.8066 at 90% coverage, which is enough to reproduce its EXECUTE/ASK/abstain policy. Train-Jev publishes per-type temperatures in calibration.json and an ECE that holds only in distribution. open-jev reports being over-confident by about 0.03 on out-of-distribution questions. Laya's reproduction is the clearest warning: the shipped checkpoint is under-confident, not over-confident, with a signed gap of -0.214, and a fitted temperature of 0.469 cuts ECE from 0.204 to 0.037. If you assume the wrong direction, your gate fails in the wrong direction.
A ten-minute audit
| Check | What to look for | Red flag |
|---|---|---|
| Files | Adapter, head, calibration, tokenizer, hashes | Adapter only, with the head "downloaded automatically" and no hash |
| Base and revision | Named base model and pinned revision | "Qwen" with no size or revision |
| Licence chain | Separate terms for adapter, base, data, weights | One licence line for a multi-component bundle |
| Training data | Source, teacher, filtering, label -1 handling | Synthetic data with no provenance |
| Evaluation split | Held-out rows, seed count, generator split | Same-generator validation presented as OOD |
| Comparator | Same-size, same-tuning, same-prompt baseline | A zero-shot baseline against a fine-tuned model |
| Calibration | Fitted on held-out data, direction stated | Raw confidence presented as calibrated |
| Hardware | Named GPU, batch size, concurrency | Latency with no hardware |
| Known failures | Adversarial, OOD, security, long context | No failure section at all |
| Third-party runs | Independent reproduction or board entry | Only the publisher's own numbers |
The audit need not be hostile. A card can be honest and thin: oss-jev-distill documents counts and splits but not the row schema, and oss-jev-train-src is four lines of packaging. Thin cards are fine for plumbing, inadequate for decisions that affect people.
The verdict
Trust a checkpoint when the claim, comparator, and split are named, and one artifact lets you rerun something. Prefer cards that publish failures, pinned revisions, and per-type calibration. Treat a vendor's own benchmark as a hypothesis, not a result, and treat a board entry as stronger evidence only if you know what the board measures.
The compliance evidence guide covers the documentation side, benchmark integrity covers contamination and anti-benchmaxxing, and measuring Jev covers the independent benchmark landscape. Before you trust a score, check the benchmark claims and JevBench entries, and remember that calibration is a property of the checkpoint on your data, not a line on the card.
