Can I run my own model through this suite?

Yes for engines that answer choice or noul questions with probabilities, including your own server over the system-one wire format. Very slow models are excluded from the board because they are no longer Jev-like.

What does the index number actually combine?

38 benchmarks in five areas, weighted by the square root of benchmark count with one area fixed, chance-corrected so random guessing scores zero, and coverage-adjusted so unanswered counts as wrong.

Why does setup warn about downloads and hashes?

Rebuild pins every source and must match published hashes, then import stages the suite with exclusion lists. Budget several gigabytes of downloads and working space, plus gated dataset logins.

Do I have to rerun everything after a failure or edition bump?

Run and pipeline resume from results files and retry only errored rows. A complete run under one edition can be rescored under the newer patch edition without rerunning the same rows.

What exactly does this kit score?

Only the public part of Decision Index 0.3, which is 37 benchmarks across five areas. The board's headline Full score is 20% public benchmarks, 50% private tests of the same skills and 30% private decision tasks from new domains, so most of the ranking comes from tests nobody can train on. Area weights are Knowledge and Reasoning 25.8%, Language Understanding 25.8%, Retrieval and Classification 20.0%, Tools and Automation 18.3%, Arts and Human Taste 10%.

Can the whole run happen on my own machine or as one job?

Yes to both. One pipeline command runs the full suite on your machine and scores it at the end, uploading nothing unless you pass --upload. Or a single Hugging Face Job on one RTX PRO 6000 uploads results.jsonl.gz, benchmark-summary.json, index.json, scores.json, environment.json and status.json to a dataset repo you own, reading the suite from a private dataset.

What keeps a model off the leaderboard entirely?

Latency. Models whose median, mean or 80th-percentile latency is over 1,000 ms per request on one RTX PRO 6000, measured single-process one request at a time, are not added, because at that speed they are no longer Jev-like. Your own scores.json latency is a guide, not that measurement. Ranks are also shared when Full scores sit under 0.9 points apart.

CommunityJev Practices & Patterns

Decision Index

Reproduction kit that rebuilds a frozen typed-decision benchmark suite from public sources and runs any engine through it with resume, scoring, and index computation.

See highlights

Open github.com

Stars: 39
Language: Python
License: MIT
Published by: Community
Author: apolinario
Last verified: Oct 2026

apolinario/decision-index

User repository on GitHub

View on GitHub

Decision Index: reproduce the Jev decision-model benchmark suite locally or as one HF Job

Forks: 76
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

Reproduction kit for the Decision Index, a benchmark for typed decision engines: models that take a state and a set of typed questions (multiple-choice choice or yes/no noul primitives with explicit criteria) and return one answer per question with a probability for every supplied option. It lets anyone rebuild the frozen suite from public sources, run any engine through it with checkpoint/resume, score every benchmark with the leaderboard's own scorers, and compute the index, on a laptop or as one Hugging Face…

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification Jev for Ranking Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

Rebuilds the frozen suite from pinned public sources instead of redistributing restricted benchmark data.
Runs transformers or HTTP engines locally or as a remote job, with checkpoint and resume per request.
Scores with the leaderboard scorers into benchmark, index, and combined submission files.
Applies chance correction, coverage penalties, area weights, and answerable-query filtering like the board.

Watch out

Rebuilding the suite needs large downloads, working space, and dataset logins, and suite files are hash-checked. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-rag-benchmark

Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.