Docs
CommunityJev Benchmarks & Evaluations
JEV Encoder Decision Index 0.3 Results
Elfsong's public Decision Index 0.3 runs for two encoders: JEV Ettin 400M scores 14.11 and a FLAN-T5 encoder 342M scores 18.61, each across 140,178 scoreable requests with zero runtime errors under the official runner and scorers. The repository declares no licence.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- Elfsong
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
- Languages
- Python
Highlights
- Two complete runs: JEV Ettin 400M at Public Index 14.11 and FLAN-T5 encoder 342M at 18.61.
- Each run covers 140,178 scoreable requests with 0 runtime errors; 140,620 requests before 442 exclusions.
- Scores come from the official runner and scorers at commit f8552a13b643d39f61185f5951bb5fed16bedf51.
- One H100 80GB per model, one sequential request at a time, at most two questions per forward pass.
- Both runs are complete, pass all suite hashes, include HLE, and their pinned model revisions are downloadable.
Quickstart
python
from huggingface_hub import snapshot_download
snapshot_download(
"Elfsong/jev-decision-index-20261013-v0",
repo_type="dataset",
revision="<commit from submission PR>",
local_dir="jev-submission",
)Watch out
Card reviewed 2026-10-17; the digest declares no licence for this results repository. These are public-suite scores, not leaderboard Full scores, H100 latency is diagnostic, and the official RTX PRO 6000 latency and private Full score are determined by the maintainers. Benchmark prompts, source datasets and training corpora are not redistributed, and the snapshot example carries a placeholder revision.
Questions about this entry
Short answers drawn from the linked project or page.
Elfsong's complete public Decision Index 0.3 evaluations for two encoder models, produced with the official runner and scorers at commit f8552a13b643d39f61185f5951bb5fed16bedf51, with preserved scoring reports, prediction records and inference instructions.
JEV Ettin 400M reached a Public Index of 14.11 and the JEV FLAN-T5 encoder 342M reached 18.61, both across 140,178 scoreable requests with zero runtime errors. Both submitted runs have complete: true and include HLE.
These are public-suite scores, not leaderboard Full scores. The card says the benchmark maintainers independently review results, run private tests and check latency, so the Full score and official RTX PRO 6000 latency are determined by them, while H100 latency here is diagnostic.
One H100 80GB per model, one sequential request at a time, at most two questions from the same request per encoder forward pass, FP32 weights with BF16 autocast, four CPU threads, 7,999 tokens per serialized question and 512 choices without truncation or filtering.
Temperatures were fitted on a reserved calibration split before benchmark evaluation, per the card. Each run's original environment.json and the inference README carry all settings and capacity limits, including pinned model revisions and exact temperatures.
Each run directory keeps original scoring reports and compact prediction records; results.jsonl.gz decompresses to the exact evaluated result bytes, untouched. A run covers 140,620 requests before the 442 official exclusions, and re-scoring uses the official toolkit.
Benchmark prompts, source datasets, training corpora and training logs are not redistributed, and the digest declares no licence for this repository; the snapshot_download example also still carries a placeholder revision from the submission PR.
More Jev resources like this
Complete Decision Index 0.2.1 runs of AutoTrust's JEV typed-decision models: engine outputs for JEV-9B, JEV-27B and JEV-Gemma4-26B-A4B, each covering 150,317 scoreable requests with zero errors.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPythonEvaluation artifacts for hotchpotch's Bekko System One v0 models on Decision Index 0.2.1: local strict, no-truncation runs scoring 7.37 (17M), 12.83 (68M) and 21.87 (400M), with predictions and environment hashes.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.