Docs
CommunityJev Benchmarks & Evaluations
Bekko System One v0 Decision Index 0.2.1 Results
Evaluation artifacts for hotchpotch's Bekko System One v0 models on Decision Index 0.2.1: local strict, no-truncation runs scoring 7.37 (17M), 12.83 (68M) and 21.87 (400M), with predictions and environment hashes.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- hotchpotch
- Last verified
- Oct 2026
Highlights
- Decision Index 0.2.1 scores: 7.37 for 17M, 12.83 for 68M and 21.87 for 400M, with raw indices 29.13, 33.24 and 39.80.
- Each run answered 150,309 scoreable requests with zero errors and 8 unsupported cases; 442 excluded records are kept for provenance.
- Median/p95 latency: 6.8/16.5 ms (17M), 11.8/30.6 ms (68M) and 17.1/72.2 ms (400M) on one RTX 5090.
- Unsupported requests split as ToolRet 5, ContractNLI 2 and POP909-CL 1; native limits are 7,999 context, 7,997 query and 3,800 document tokens.
- The card says these local strict runs are not official leaderboard scores and await maintainer validation and training-overlap policy review.
Quickstart
bash
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python --extra-index-url https://download.pytorch.org/whl/cu130 --index-strategy unsafe-best-match -r code/requirements.lock
uv pip install --python .venv/bin/python --no-deps ./code/decision-indexWatch out
Card reviewed 2026-10-17; the card stresses these are local strict/no-truncation results pending maintainer validation and training-overlap policy review, not official leaderboard scores, and that the original scores apply no training-overlap correction. Prediction artifacts confer no new licence and the suite itself is not redistributed.
Questions about this entry
Short answers drawn from the linked project or page.
Predictions, scores, runtime provenance and training-overlap disclosures from Bekko System One v0 runs (17M, 68M and 400M) on Decision Index 0.2.1; it does not redistribute the benchmark inputs themselves.
7.37 for 17M, 12.83 for 68M and 21.87 for 400M, with raw indices 29.13, 33.24 and 39.80. The card notes DI is chance-corrected and coverage-adjusted, not accuracy.
No. The card describes them as local strict/no-truncation results pending maintainer validation and training-overlap policy review, and says the original scores apply no training-overlap correction.
With the official Decision Index reproduction kit on one RTX 5090, PyTorch 2.10.0+cu130, one runner process per request, no benchmark-specific prompt tuning, option filtering or truncation, and adapter hashes recorded in environment.json.
Median and p95 request times are 6.8/16.5 ms for 17M, 11.8/30.6 ms for 68M and 17.1/72.2 ms for 400M; the card says local timings do not substitute for the maintainer's held-out parity and latency check.
The repo pins the reproduction kit at commit 87d4650b42b377c0291a89c1f1a879f9b31082bf and includes a CPU-only re-score path plus a GPU inference path through code/reproduce.py, though rebuilding the suite needs upstream access including HLE.
Prediction artifacts confer no new licence on the suite or model sources; the adapter and vendored kit carry their own repository licences and upstream model and training-source terms remain applicable.
More Jev resources like this
ML Intern Lab's benchmark write-up of open decision models on the LocalLLaMA/typed-decisions test split: 2,000 decisions per model for GEV-26B-Decide, JEV-35B and pplx-decider-v1.1-27b, with accuracy, KL, Brier, ECE and latency tables. The repository states no licence.
Jev Benchmarks & EvaluationsDocs#community#evaluation#benchmark
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPythonjohannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.