What does the rejection-policy benchmark test?

Whether rejection policies remain useful as candidate sets and tasks change, comparing Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets.

Does rejection calibration transfer across tasks?

Source calibration often fails to preserve the target operating point; a Jev policy calibrated on DBpedia rejects 69.3% of covered Emotion test inputs, while an Emotion policy loses detection entirely.

What retrieval tradeoff does the benchmark expose?

With ten intent candidates, Laya detects 99.0% of out-of-scope queries but rejects 48.8% of covered queries.

Which missing-answer sources are separated?

Three: artificial omission, natural retrieval misses, and public out-of-scope queries, so their costs can be read alongside retrieval coverage.

What changed in version two of the benchmark?

It expands the evaluation with Qwen2.5-7B-Instruct and CLINC150, adds fixed-budget rejection policy transfer, three missing-answer sources, and controlled robustness analyses.

Where are the rejection benchmark artifacts?

Code and benchmark artifacts are available at github.com/luckykevvv/Decision_Model_Benchmark.

What labels does the rejection evaluation use?

Public reference labels, with shared inputs, explicit decision and failure categories, and reproducible scoring for assessing rejection policies under the conditions in which they are reused.

CommunityJev Benchmarks & Evaluations

Benchmarking candidate coverage and rejection policy transfer in typed decision models (arXiv 2610.03387)

A benchmark of rejection-policy transfer across Laya, Jev, and Qwen2.5-7B-Instruct at equal calibration budgets, covering artificial omission, natural retrieval misses, and public out-of-scope queries. Source calibration often fails to preserve the target operating point.

See highlights

Open arxiv.org luckykevvv/Decision_Model_Benchmark

Stars: 0
Language: Python
License: MIT
Format: Article
Published by: Community
Last verified: Oct 2026

luckykevvv/Decision_Model_Benchmark

User repository on GitHub

View on GitHub

Benchmark of Decision Model for arxiv paper "Benchmarking candidate coverage and rejection policy transfer in typed decision models"

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity research arxiv rejection benchmark calibration

Works forJev for Classification Jev for Decision gating Jev for Evaluation

Used inJev for Software & DevTools Jev for Data & Analytics

Languages: Python

Highlights

Compares Laya, Jev, and Qwen2.5-7B-Instruct using public reference labels at equal calibration budgets.
A Jev policy calibrated on DBpedia rejects 69.3% of covered Emotion test inputs; an Emotion policy loses detection entirely.
With ten intent candidates, Laya detects 99.0% of out-of-scope queries but rejects 48.8% of covered queries.
The benchmark separates artificial omission, natural retrieval misses, and public out-of-scope queries.
Code and benchmark artifacts are available at github.com/luckykevvv/Decision_Model_Benchmark.

Watch out

arXiv preprint posted October 2, 2026 and updated October 6, 2026 (cs.AI, cs.CL; 29 pages) by Jiawen Lu and Tongtong Wu. The v2 expansion adds Qwen2.5-7B-Instruct and CLINC150, and the transfer results depend on the specific source and target pairs tested.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

More Jev resources like this

TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models (arXiv 2610.11392)

A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

GraphDecisionBench: Benchmarking System One Models on Graph Tasks (arXiv 2610.06354)

A model-independent benchmark for graph decision tasks, combining structural task profiles, matched graph-text input contrasts, and heuristic-proposal controls. It finds accurate adjacency recognition does not guarantee broader structural correctness.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

VictorYXL/GraphDecisionBenchPython

Evaluating and Benchmarking the System One Model Jev (arXiv 2609.37647)

A zero-shot evaluation of Jev (jev-1.13.0) on 37 datasets spanning classification, routing, NLI, reading comprehension, commonsense reasoning, moderation, legal clause analysis, and rubric scoring. It runs 346,009 requests for under USD 10 and reaches 95–99% accuracy on IMDB, SST-2, HellaSwag, and ARC.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

AppliedMachineLearning-Lab/jev-benchmarkingPython

Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration (arXiv 2609.35342)

Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

What the Jev

A reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.

Jev Benchmarks & Evaluations#community#python#evals

Community

keta1930/what-the-jevPython

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models (arXiv 2610.18610)

A benchmark evaluating Theory of Mind selection in typed decision models against generative baselines: 11,117 English questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy and ranks first or second on every capability.

Jev Benchmarks & EvaluationsArticle#community#research#arxiv

Community

HKUST-KnowComp/Jev-ToMPython

From the Jev guides

Original write-ups on what this entry is for.

What the First Month of Jev Papers Found

Oct 14, 2026

More than eighty preprints now evaluate, attack, and reproduce the typed-decision class. The broadest board, the coherence gap, the rejection bottleneck, and the attacks that still work.

Jev Practices & Patterns#research#evaluation#calibration

Read article

Designing for Rejection

Oct 14, 2026

A Choice can only pick from its options. How rejection options, candidate coverage, and escalation bands keep a narrow taxonomy honest under inputs it never anticipated.

Jev Practices & Patterns#rejection#confidence#patterns

Read article

Measuring Jev: What Independent Benchmarks Actually Found

Sep 22, 2026

TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.

Jev Practices & Patterns#benchmarks#evaluation#calibration

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai