Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Evaluation & TrainingError matrix

Confusion matrix

The table of predicted versus actual labels that shows which classes a model actually confuses, not just how often it is right.

Open docs.typesafe.aiAll 164 terms
Category
Evaluation & Training
Also known as
Error matrix
Related terms
5
Directory entries
55
Docs
docs.typesafe.ai
Added
2026-10-12

Definition

A confusion matrix crosses predicted labels with true labels, so the diagonal is correct and the off-diagonal cells are the specific mistakes. Aggregate accuracy hides the structure the matrix reveals: a model can be excellent on the common classes and terrible on a rare one and still look fine.

For a Choice question it is the natural read-out, because the answer is already one of your options. Read the worst off-diagonal cell first and ask whether the two options' descriptions are distinguishable, the criterion behind the contrastive-criteria advice.

Tagsevaluationbenchmarks

Related terms

Definitions that connect to this one.

Precision and recallThe two error-facing rates of a classifier: how often a yes is right, and how often a real yes is caught, plus the F1 that blends them.Evaluation & TrainingBenchmark claimsTypeSafe's headline multipliers — 193.6x faster, 444.6x cheaper, a 238x lower input price — come from its own workflow comparisons.Evaluation & TrainingCalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceHuman reviewThe explicit path for uncertain or high-stakes cases: route to a person instead of acting on a low-confidence answer.PatternsConsensus labelsReference answers built from two frontier models at high thinking, used as the ground truth in TypeSafe's workflow evals.Evaluation & Training

From the Jev directory

Jevals — Independent Jev Benchmark for Models That Decide
An independent benchmark of hosted Jev against six LLMs on PubMedQA, Banking77, and HelpSteer2, scoring decisions against human labels and reporting per-board results for each model.
Jev Practices & PatternsArticle#community#benchmarks#evaluation
Community
Best Decision Model APIs in 2026
A ranked comparison of thirteen decision-model APIs as of October 2026, including Jev, OpenAI Decisions, Clef, Liquid d1, Kev, Perplexity Decider, and Strands, scored on evidence, usage, and price.
Jev Benchmarks & EvaluationsArticle#community#comparison#decision-model
Community
A decision model d1 surpasses Jev
GIGAZINE's report on Liquid AI's d1, which it describes as the first decision model to beat Jev on Hugging Face's Decision Index, and a short survey of the clone wave that followed Jev's September launch.
Jev Media & TalksArticle#community#news#benchmarks
Community
Typed Decisions Benchmark
The 400-case, 2,000-decision benchmark open models train and report against: four workflows, state plus typed questions, and gold distributions averaged from three teacher samples.
Jev Benchmarks & EvaluationsDocs#community#benchmarks#evaluation
Community
Typed Decision Leaderboard
An independent 2,018-item answer-verifier benchmark: the same items and labels for every entrant, each run on its own maker's code, with Jev, ZTC-Judge, Darwin, and open models ranked by weighted AUC.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community
S1MB Leaderboard
The System One Mosaic Benchmark leaderboard: about 100 decision models compared across 100+ specialized tasks, with results regenerated from a public dataset rather than estimated.
Jev Benchmarks & EvaluationsPlayground#community#evaluation#benchmarks
Community

49 more matching entries in the full directory.

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai