Skip to content
JevDirectory.orgTraffic count unavailable. View site traffic
Evaluation & TrainingCoverage

Candidate coverage

The share of real inputs for which the option list actually contains the right answer, measured separately from the model's accuracy at picking it.

All 172 terms
Category
Evaluation & Training
Also known as
Coverage
Related terms
4
Directory entries
1
Docs
—
Added
2026-10-14

Definition

A typed decision can only choose from the options it was given, so two failure sources hide inside one accuracy number: the model picking wrong from the list, and the list itself not containing the answer. Candidate-coverage work separates them by testing artificial omissions, retrieval misses, and out-of-scope queries, and by scoring rejection behavior under each.

The benchmark that introduced the measure found rejection policies do not transfer: a Jev policy calibrated on DBpedia rejected 69.3% of covered Emotion test inputs, while an Emotion policy lost detection entirely. Coverage is therefore a property to measure per candidate set and per task, before trusting a threshold tuned somewhere else.

Tagsevaluationresearch

Related terms

Definitions that connect to this one.

Rejection optionAn explicit choice like other or none-of-the-above that gives the model a way to decline, and the reason an option list without one can only pick the least-wrong answer.PrimitivesOracle floorThe best error rate achievable with perfect choices from the available options, used to judge how close a system gets.Evaluation & TrainingThresholdThe value your code compares against a score, a Noul probability, or a confidence to turn an answer into an action.Answers & ConfidenceDistribution shiftWhen the traffic a model sees at run time stops looking like the data it was calibrated on, so its probabilities no longer mean what they did.Evaluation & Training

From the Jev directory

Benchmarking candidate coverage and rejection policy transfer in typed decision models (arXiv 2610.03387)
0GitHub stars
A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Communityluckykevvv/Decision_Model_BenchmarkPython

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai