Docs
CommunityJev Benchmarks & Evaluations
Typed-Decisions Benchmark Results
ML Intern Lab's benchmark write-up of open decision models on the LocalLLaMA/typed-decisions test split: 2,000 decisions per model for GEV-26B-Decide, JEV-35B and pplx-decider-v1.1-27b, with accuracy, KL, Brier, ECE and latency tables. The repository states no licence.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- ML Intern Lab
- Last verified
- Oct 2026
- Languages
- Shell
Highlights
- Large-model run: 400 cases x 5 typed questions = 2,000 decisions per model on one A100-80GB.
- Accuracy is a photo finish: GEV 0.7210, JEV-35B 0.7295, pplx-decider-v1.1-27b 0.7315, all under the 0.735 ceiling.
- GEV-26B-Decide leads calibration and speed: KL 0.322, Brier 0.123, ECE 0.028 and a 40 ms p50.
- Small-specialist rerun on one T4: laya-typed-decisions 0.767 accuracy with Brier 0.062; FRIDA-Decisions 0.406.
- All 2,000 decisions per model returned valid distributions with 0 errors in both runs.
Quickstart
bash
hf download autotrust/GEV-26B-Decide # or autotrust/JEV-35B
bash <model_dir>/serve.sh # GEV: apply scripts/apply_gemma4_patch.py to vllm first
python run_eval_server.py --url http://127.0.0.1:8000 --model-name <repo-id> --out-prefix out
python run_eval_pplx.py --out-prefix outWatch out
Card reviewed 2026-10-17; the digest declares no licence for this results repository. Latency is single-request client-side on an idle A100 rather than a serving benchmark, the hosted leaderboard uses a different request protocol, GEV ran with thinking off and a content-based port of the repo's lm_head-LoRA patch, and gold is a soft three-sample teacher distribution.
Questions about this entry
Short answers drawn from the linked project or page.
ML Intern Lab's benchmark results for open decision models on the LocalLLaMA/typed-decisions benchmark. Two runs are published: a large-model run on one A100-80GB from 2026-10-13, and an earlier pass over small specialist models on a single T4.
Config all, test split, 400 cases x 5 typed questions = 2,000 decisions per model, one decision per request, sequentially with client-side latency. Each model ran through its native interface: GEV and JEV via bundled serve_decide.py on vLLM 0.31.0, pplx through its bundled DecisionModel.
Accuracy is a three-way photo finish: GEV-26B-Decide 0.7210, JEV-35B 0.7295, pplx-decider-v1.1-27b 0.7315. GEV wins calibration and speed: KL 0.322, Brier 0.123, ECE 0.028 and a 40 ms p50 against 113 ms and 142 ms.
Per-workflow accuracy disagrees: GEV is weakest on agent_trace at 0.612 and strongest on customer_service and invoice_processing, JEV tops customer_service at 0.812, and pplx leads security_incidents at 0.750 and score questions at 0.695. Accuracy by question type clusters within a point.
From the LocalLLaMA/typed-decisions card: a Prior baseline that ignores input scores 0.470, the teacher self-agreement ceiling is 0.735, and hosted TypeSafe Jev 1.13 reports 0.727 zero-shot under a different request protocol, so it is not directly comparable.
On one T4 in fp32: laya-typed-decisions (421M) reached 0.767 accuracy with Brier 0.062 at 122 ms per case, embeddinggemma-2-ft (272M) 0.764 at 670 ms, and FRIDA-Decisions (823M) 0.406, near the 0.318 random guess. The first two land almost exactly on their published numbers.
Latency is single-request client-side on an idle A100, so it is relative speed, not a serving benchmark; GEV ran with thinking off and a ported lm_head-LoRA patch for the vLLM 0.31.0 release. Gold is a soft three-sample teacher distribution, and the repository declares no licence.
More Jev resources like this
Evaluation artifacts for hotchpotch's Bekko System One v0 models on Decision Index 0.2.1: local strict, no-truncation runs scoring 7.37 (17M), 12.83 (68M) and 21.87 (400M), with predictions and environment hashes.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Real human-labeled data reformatted into System One questions: 22 configs, 166,054 rows and 22,773 test records where choice, score and noul answers are scored as probability distributions.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
A benchmark for typed decision models such as Jev, built from seven policy-labelled generators and nine evaluation suites. It reports accuracy as medians across paraphrases, calibration against a matched noise floor, and realized cost under asymmetric cost matrices.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
Introduces Sys1Cal-v1, a dataset of True/False questions about propositions whose exact probability is known by construction, queried through Jev's three primitives and scored by total variation from ground truth. Recovering a suppressed third value raises median soft accuracy from 0.771 to 0.978.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.