The Decision Index is less a leaderboard than a panel of instruments sharing a name. One publishes complete engine outputs for a fixed suite. Another ranks hosted APIs. A third tests whether probabilities stay coherent when questions are reworded. A fourth scores vendor-written workflows against labels from two frontier models. A rank is a claim about a suite, recipe and scoring rule, not about your task.
A panel, not one board
The community Decision Index results publish complete engine runs; the board resolves near-ties with a 0.9-point rule. In the System One Models roundup, Jev sits third of 117 rows at 60.11, behind pplx-decider v1.1 at 62.75 and Fastino GLiDE at 60.21. The tie rule gives Jev, GLiDE and Torchcast Decision 27B a shared second place. The separate Artificial Analysis Decision Index ranked Jev first of 11 hosted decision models at 57.5 at launch; after Liquid's d1-ultra was added, Jev was second of 13.
| Panel | Unit being scored | What the rank means |
|---|---|---|
| Community Decision Index | Complete engine runs on a shared suite | Relative score on that suite, with a tie rule |
| Artificial Analysis Decision Index | Hosted decision APIs | Relative rank among the models entered |
| AutoTrust run repositories | One model, one recipe | Reproducible outputs for a named configuration |
| JevBench | 534 frozen cases per entrant | Geometric mean of intelligence, calibration, speed and cost |
| JevBench Metamorphic | 2,240 questions under 50 rewrites | Coherence, not correctness |
| TypedBench | Paraphrase families and policy suites | Median accuracy, calibration against a noise floor, cost |
What a run repository actually contains
The AutoTrust repository holds three runs: JEV-9B at a public index of 43.14 and a raw score of 56.91, JEV-27B at 53.30 and 64.32, and JEV-Gemma4-26B-A4B at 58.05 and 67.35. Each covers 150,317 of 150,317 scoreable requests with zero errors. The hardware was one NVIDIA B200; sequential median latency was 77 ms.
The caveats are in the details. Only engine outputs are published, as compact rows with ids, statuses, answers, probabilities and timings; the inputs are not included, so the suite must be rebuilt with the index kit. The batched phase reports total_wall_ms as batch wall time divided by 64 rather than per-request latency. The JEV-27B run repeats the same recipe on Qwen/Qwen3.8-27B. The decision head has 16 choice slots, so a question with more than 16 options needs ceil(n/16) + 1 passes.
The encoder repository is a second check. Elfsong's 0.3 runs score JEV Ettin 400M at 14.11 and a FLAN-T5 encoder at 18.61, each across 140,178 scoreable requests with zero runtime errors, using the official runner at a pinned commit. The card says these are public-suite scores, not leaderboard Full scores, that H100 latency is diagnostic, and that the official RTX PRO 6000 latency and private Full score are set by the maintainers.
Individual run records are similarly granular. The Kieve-JeV Fractal 300M results separate RouterBench 0-shot and 5-shot, GSM8K 10-choice and 4-choice, and five iSarcasmEval splits. RouterBench and GSM8K carry numeric scores; the iSarcasmEval tracks are null while their scored request counts remain. The record preserves Brier score, log loss, nDCG@10, MRR, candidate recall, cost utility, utility regret, and median and p95 latency.
The gap between a raw score and a rank
Check which score is quoted. The AutoTrust runs publish both a raw score and a public index: 56.91 versus 43.14, 64.32 versus 53.30, and 67.35 versus 58.05. The public index is the comparable number; the raw score is not a drop-in substitute.
Check the recipe. The 27B entry applies the same recipe to a Qwen checkpoint, which makes it a test of the recipe as much as of the model family. The encoder runs pin a runner commit and a calibration split, but they are still public-suite results. Neither repository is a private Full evaluation, and the maintainers run those separately.
Check the missing half. Public rows can be trained on or selected against, and the private Full score is not in the artifact. Read the panel as a screening instrument.
What the other instruments add
JevBench runs 534 frozen cases per complete entrant, including 220 hard decisions written by Claude Opus 5 and GPT-5.6 Sol and hashed before any system ran. Its score is the geometric mean of chance-corrected intelligence, calibration, speed and cost, 25% each. Jev 1.13.0 ranks second at 74.4 behind Hopper at 75.4, reading 950 input tokens per decision. Cost is dollars per 1,000 decisions: Jev's $0.0399 comes from 950 tokens at $0.042 per million. The harness also documents option-order sensitivity: reversing A and B dropped open-alternative-jev from 72% to 21% on answer judging.
JevBench Metamorphic uses 240 synthetic English cases, 20 in each of 12 domains, with 2,240 questions and 50 relations in five dimensions. The full run executes 12,000 tests and 10,320 requests; the mini run 1,200 tests and 1,248 requests. The rewrite bank holds 823 accepted rewrites from 1,697 candidates. Coherence is not correctness: a model that answers uniformly scores higher than every evaluated model, so the number must be read beside accuracy and the Brier score.
TypedBench adds the statistical frame. It is built from seven policy-labelled generators and nine suites, reports accuracy as a median and range across paraphrases, and judges calibration against the finite-sample noise floor of a matched, perfectly calibrated predictor. Its hosted model follows the stated policy but is wording-sensitive and systematically underconfident, and under asymmetric costs using its probabilities can be worse than taking its top answer.
Where the vendor numbers sit
TypeSafe's workflow evals score four workflows against consensus labels from GPT-6 Astra and Claude Fable 5.1 at high thinking. Jev averages 67.8% accuracy at $0.0004 and 0.4 s per case. The labels are not ground truth and the harness is vendor-run and assumed correct. The page also shows every model is more accurate, cheaper and faster in a workflow than with the same policy as a prompt.
The jev-ai.org audit puts those numbers beside the alternatives: 67.8% for Jev against 74.1% for "sol" at $0.0836 and 23.3 s, with Jev about 76 times cheaper and 25 times faster than "terra," or 8 times cheaper and 32 times faster than "luna." It notes the weakest workflow, invoice processing, where Jev scored 61.8% against 79.1% for sol, and that the homepage demo is one short dense query TypeSafe itself says flatters Jev.
The 37-dataset evaluation ran 346,009 requests for under USD 10. It reports 95–99% on IMDB, SST-2, HellaSwag and ARC, 86.7% on Belebele across 122 languages, and wins against Qwen3.8-27B on 27 of 37 datasets. It also reports the failure surface: all three models degrade on low-resource languages, fine-grained or noisy labels, and rubric judgments.
How to read a rank
- Check the suite. Public-suite, Full and vendor-workflow scores are not interchangeable.
- Check the recipe and checkpoint. A run on a Qwen base is not a run on the original model.
- Check the metric. Accuracy, calibration, Brier, cost and latency can rank the same systems differently.
- Check the labels. Consensus from two frontier models is not ground truth.
- Check what is missing. No inputs, no licence, or a batched latency field can make a run hard to reproduce.
The panel is useful the way a flight instrument panel is useful: several readings constrain one another. The decision that matters is still the one measured on your own questions, which is why the next step is the fit-first work in measuring Jev and the survey in what the papers found. For engine selection, the trade-offs are closer to choosing an open decision model. See the Decision Index, leaderboard and calibration entries for vocabulary.
