Jev directory
CommunityJev Benchmarks & Evaluations
Your Language Model Is Already a Decision Model
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
- Stars
- 6
- Reading time
- 2 min
- Published by
- Community
- Added
- Oct 2026
Highlights
- Compares untrained Qwen3.5-9B with hosted Jev on decision, tool-calling, and calibration benchmarks.
- Qwen uses next-token probabilities with no fine-tuning, LoRA, or trained decision head.
- Qwen runs on one NVIDIA H100 80GB GPU per worker; Jev is accessed through its hosted API.
- Jev leads Typed Decisions 73.9% to 62.1% and JevBench 86.1% to 80.5%.
- Qwen leads PhishNChips 73.9% to 62.5% and MetaTool Task1 83.4% to 77.1%.
Questions about this entry
Short answers drawn from the linked project or page.
Use the provided command with CUDA_VISIBLE_DEVICES=0 QWEN35_USE_FLA=1 models/open_jev_py311/bin/python scripts/serve_qwen35_jev.py --device cuda:0 --port 8796 --max-length 8192 --fallback-margin 0.
Qwen runs on one NVIDIA H100 80GB GPU per worker; Jev is accessed through its hosted API.
It returns the expected answers for Choice, Noul, and Score while identifying the actual model as qwen3.5-9b.
It accepts Jev's POST /v1/systemone JSON interface with state and typed questions, and returns the expected answers object.
On six matched labeled benchmarks with 5,212 decisions, Qwen's mean 10-bin ECE is 0.055 versus 0.099 for hosted Jev; mean multiclass Brier scores are 0.323 and 0.322.
Qwen leads on PhishNChips at 73.9% versus 62.5% and MetaTool Task1 at 83.4% versus 77.1%; BFCL and WebShop results are nearly identical.
Jev leads on JevBench, Typed Decisions, WebPRM, and When2Call, including Typed Decisions at 73.9% versus 62.1% and WebPRM full at 66.6% versus 59.0%.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Hugging Face dataset of 130k rows pairing RAG claim-verification and response-quality questions with Jev-1.13.0 predictions, gold labels, confidences, token counts, latency, and cost. Derived from wandb/RAGTruth-processed.
Jev Benchmarks & Evaluations#evaluation#classification#benchmark
Community
Controlled benchmark of Jev against Claude Sonnet 5 on one coding-agent decision: given a bug report, which file needs editing? It scores 200 real SWE-bench Verified issues with cost, latency, and confidence-band routing tables.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
CommunityK624/jev-localization-bench
Spring Boot and Spring AI benchmark comparing Java keyword rules, Jev 1.13.0, and Claude Haiku 5.5 on warehouse goods-receipt triage, with a 200-remark synthetic dataset and a Python analysis script.
Jev Benchmarks & Evaluations#benchmark#evaluation#calibration
Communitymohammedahmadi1990/jev-triage
A measurement lab for TypeSafe AI's Jev, run against api.typesafe.ai/v1/systemone with model jev-latest. It records baseline confidence scores, position-bias controls, atomic-question failures, and a 24-README semantic triage sweep, with committed data and analysis scripts.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
CommunitySuperInstance/jev-lab
ML Intern Lab's benchmark write-up of open decision models on the LocalLLaMA/typed-decisions test split: 2,000 decisions per model for GEV-26B-Decide, JEV-35B and pplx-decider-v1.1-27b, with accuracy, KL, Brier, ECE and latency tables. The repository states no licence.
Jev Benchmarks & EvaluationsDocs#community#evaluation#benchmark
Community
A 41,500-response bias audit of Jev across cognitive reflection, five bias paradigms, and a pricing experiment: selected answers look robust, but probabilities move with irrelevant numbers, option order, and framing.
Jev Benchmarks & EvaluationsArticle#community#research#arxiv
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.