What does DecisionBench measure exactly?

It evaluates document-grounded decision models on Noul, Choice and Score answers from pinned tasks, counting malformed probability distributions as misses so comparisons remain reproducible.

Can I add my own model or task to DecisionBench?

Yes. The docs cover adding a model adapter, a dataset-backed task, a benchmark collection or reviewed result records; code changes go to the repository that owns that component.

How large is the suite, and what is in it?

23,900 rows spread across nine families and three primitives, plus a separate reasoning track. Results are broken out by task, family, domain and primitive on the leaderboard, so you can filter rather than read a single aggregate.

How do I install it and run a smoke test?

Either `pip install git+https://github.com/Hanno-Labs/decision-bench.git` or `uv add` from the same git URL. Then download a pinned model with hf download, passing a --revision, and run `decision-bench run-hf task_specs/decisionbench-dev.toml --smoke` against the pinned dev task spec.

Do I need to write an adapter for my model?

Only if its native decision readout is not already supported. The docs list the supported adapters and models, and the guidance is to use the existing runner when your model is listed. Native decision-token readouts are supported directly, for example Hanno-Labs/bosun-v3.1-0.6b through run-hf.

Where do results live, and how current is this?

The leaderboard is a Hugging Face Space that compares reviewed results, and reviewed result records live in a separate results repository. Comparability depends on pinned revisions and reviewed submissions, and the project is under active development with the benchmark paper not yet published.

What license is it under?

Apache-2.0, with CI running on every change. Bugs and requests covering any component go to the central issue tracker, while code changes go to the repository that owns that component.

CommunityJev Benchmarks & Evaluations

DecisionBench

An open benchmark runtime for document-grounded decision models: it scores Jev Noul, Choice and Score answers on pinned tasks and counts malformed probability distributions as misses so model comparisons stay reproducible.

See highlights

Open github.com

Stars: 3
Language: Python
License: APACHE-2.0
Published by: Community
Last verified: Oct 2026

Hanno-Labs/decision-bench

Organization repository on GitHub

View on GitHub

Open benchmark runtime for document-grounded decision models

Forks: 2
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

The evaluation ecosystem for decision models. Installation · Documentation · Leaderboard · Results · Issues · Citing. Smoke-test a supported decision model on the pinned benchmark. This example uses Bosun v3.1 0.6B, whose native decision-token readout is supported directly by run-hf. Before running anything else, see the complete supported adapters and models. If your model is listed, use its runner; only add an adapter when its native decision readout is not already supported.

Quoted from the project's README. Read the full README.

Tagscommunity benchmark evaluation decision-models

Works forJev for Evaluation

Used inJev for Data & Analytics

Languages: Python
Platforms: CLI
Capabilities: Open source

Highlights

Task suite described as 23,900 rows across nine families and three primitives, with a separate reasoning track.
Installable with pip or uv from git; decision-bench run-hf runs a smoke test on the pinned dev task spec.
Leaderboard is hosted on Hugging Face Spaces, with reviewed results in a separate results repository.
Apache-2.0; supports native decision-token readouts such as Hanno-Labs/bosun-v3.1-0.6b.

Watch out

Under active development with the benchmark paper not yet published; comparability depends on pinned revisions and reviewed submissions. Apache-2.0 licensed.

Questions about this entry

Short answers drawn from the linked project or page.

Related terms

Glossary definitions related to this entry.

Precision and recall

More Jev resources like this

jev vs. open alternatives

Notebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.

Jev Benchmarks & Evaluations#community#evaluation#document-processing

Community

run-llama/jev_vs_ossJupyter Notebook

DeepSearcher Jev stopping experiment

8.3k

Agentic search evaluation inside Zilliz DeepSearcher: a standalone harness replaces the ChainOfRAG reflection call with one Jev Noul per round to decide whether to stop within a seven-round budget, then compares stopping behavior, evidence recall and decision cost on 100 sampled 2WikiMultiHopQA queries.

Jev Benchmarks & Evaluations#community#rag#evaluation

Community

zilliztech/deep-searcherPython

hev reranker

A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.

Jev Benchmarks & Evaluations#community#evaluation#python

Community

hev/rerankerPython

System-One Control Bench

A benchmark that steers decision models through 100 key-and-door grid puzzles with one Jev Choice per move, scored against an exact BFS solver; it found Jev picks an optimal move about 90% of the time yet finishes only 57 of 100 games.

Jev Benchmarks & Evaluations#community#benchmark#gridworld

Community

Exemocaro/System-One-Control-BenchPython

S1MB

System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.

Jev Benchmarks & Evaluations#community#benchmark#leaderboard

Community