Docs
CommunityJev Benchmarks & Evaluations
JEV Decision Index Results
Complete Decision Index 0.2.1 runs of AutoTrust's JEV typed-decision models: engine outputs for JEV-9B, JEV-27B and JEV-Gemma4-26B-A4B, each covering 150,317 scoreable requests with zero errors.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- AutoTrust AI
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
- Languages
- Python
- Capabilities
- Open source
Highlights
- Three complete runs: jev-9b at 43.14, jev-27b at 53.30 and jev-gemma4-26b-a4b at 58.05, with raw scores 56.91, 64.32 and 67.35.
- Each run covers 150,317 of 150,317 scoreable requests with 0 errors and 0 unsupported cases.
- Published per run: compact result rows, scores.json, index.json, environment records, engine source with sha256 and a training-overlap check.
- Hardware: one NVIDIA B200 (183 GB) with torch 2.13.0, CUDA 13.0, transformers 5.16.1 and peft 0.21.0; sequential median latency 77 ms.
- The overlap check reports no whole suite state, instruction or option in the training data.
Watch out
Apache-2.0, but only engine outputs are published as compact result rows without benchmark inputs, so the suite must be rebuilt with the index kit. The batched phase reports total_wall_ms as batch wall time divided by 64 rather than per-request latency, and the jev-27b run repeats the same recipe on Qwen/Qwen3.8-27B.
Questions about this entry
Short answers drawn from the linked project or page.
Decision Index 0.2.1, built with the decision-index kit and published for the Jev Decision Index Space. This repository holds the complete run outputs of the JEV typed-decision models on that suite.
Three: JEV-9B at 43.14 with raw 56.91, JEV-27B at 53.30 with raw 64.32 and JEV-Gemma4-26B-A4B at 58.05 with raw 67.35, each complete on the 0.2.1 suite.
No. Only engine outputs are published, as compact result rows with run ids, statuses, answers, probabilities and timings; the card says to rebuild the suite with the kit since no benchmark inputs are included.
Rows 1–35,590 ran with the kit's sequential runner and the rest through source/fast_run.py answering 64 requests at a time; re-running 3,000 phase-1 requests through the batched driver gave 7,044 of 7,051 identical answers, the rest near-ties.
No whole suite state or question appears in the training data, with zero sentence-level hits in yuri_v3 and openjev_v2; 547 yuri_v1 rows share Wikipedia sentences but carry uniform placeholder labels.
One NVIDIA B200 with 183 GB, torch 2.13.0 with CUDA 13.0, transformers 5.16.1, peft 0.21.0 and kit revision 87d4650; sequential single-request latency is a 77 ms median.
The decision head has 16 choice slots (A–P), so a question with more than 16 options is answered in ceil(n/16) + 1 passes; the card also reports a 262,144-token context with nothing truncated.
More Jev resources like this
System One Mosaic Benchmark: a Python evaluator and Next.js viewer comparing Jev and open decision models across 137 English Choice, Noul and Score benchmarks from 106 dataset subsets, with recorded evaluation conditions and a public Hugging Face leaderboard.
Jev Benchmarks & Evaluations#community#benchmark#leaderboard
Community
hotchpotch/S1MBPython
hotchpotch/S1MBPythonElfsong's public Decision Index 0.3 runs for two encoders: JEV Ettin 400M scores 14.11 and a FLAN-T5 encoder 342M scores 18.61, each across 140,178 scoreable requests with zero runtime errors under the official runner and scorers. The repository declares no licence.
Jev Benchmarks & EvaluationsDocs#community#evaluation#benchmark
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
English, Turkish and German decision-model test data with probability targets: 3,116 held-out rows across three languages, where Deem-4B scores 94.7% in English against Kev-4B's 84.7%.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
Evaluation and training artifacts from Jev Alt System One work: teacher merge outputs, baseline summaries and Kev-0.8B plus Kev-4B typed-decision records with the scripts that built them.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.