Article
CommunityJev Benchmarks & Evaluations
Small Heads, Strong Decisions: A Controlled Study of Typed-Decision Models
arXiv paper that re-implements five readouts for typed-decision models and trains them under one protocol: the same backbone, 15 public English training sources, a per-head learning-rate sweep, and the same seed. It reports accuracy, inference, and training comparisons across Qwen3 backbones.
- Format
- Article
- Reading time
- 2 min
- Published by
- Community
- Added
- Nov 2026
Highlights
- Re-implements five readouts for typed-decision models and trains them under one protocol.
- On Qwen3-0.6B, every readout reaches 81.3-82.0% in-domain accuracy.
- A 0.5M-parameter cosine head matches a 27M-parameter joint schema head in-domain at half inference time.
- Reading all of a text's questions from one prompt matches one prompt per question in accuracy.
- Moving from Qwen3-0.6B to Qwen3-4B adds 4 in-domain points and 11 held-out points.
Questions about this entry
Short answers drawn from the linked project or page.
It re-implements five readouts for typed-decision models and trains all of them under one protocol to see which design choices affect accuracy, inference speed, and training speed.
It names Nimble, the Decision 1.0-3.0 family, and Clef as open typed-decision models that followed Jev, each pairing a different readout with a different backbone, training mix, and recipe.
All readouts used the same backbone, the same 15 public English training sources, the same recipe with a per-head learning-rate sweep, and the same seed.
No. On Qwen3-0.6B every readout reaches 81.3-82.0% in-domain accuracy, and a 0.5M-parameter cosine head matches the 27M-parameter joint schema head adapted from Clef in-domain at half its inference time.
It matches one prompt per question in accuracy, with 1.6 times faster inference and 2.1-2.5 times faster training on one H100.
More than either head choice: moving from Qwen3-0.6B to Qwen3-4B adds 4 points in-domain and 11 points on held-out sources, while base, instruct, and thinking checkpoints of the same model tie.
They recommend spending effort on the backbone and keeping the head minimal, based on the finding that head complexity did not affect accuracy in their controlled study.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Open, deterministic audit benchmark for System 1 decision models, covering calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness across five open-weight models and generative baselines.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper testing representation invariance and abstention on measured optical spectra of silicon metasurfaces and gold nanoparticles, evaluating d3-nano, Jev 1.13, three GPT-6 Luna configurations, and Qwen3.5-2B.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
arXiv paper scoring candidate pages by the citation share they will receive, comparing pretrained rerankers, Jev, eight open decision models and distilled students on GEO-Bench queries answered by ten engines. Releases GEO Citation Ranker 4B, the benchmark, all scorer predictions and code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
Benchmark comparison of an untrained Qwen3.5-9B against hosted Jev across decision, tool-calling and calibration benchmarks. Reports accuracy, latency and ECE on matched tasks, plus a local adapter that serves Qwen through Jev's typed systemone API.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
arXiv paper comparing three decision models and three general LLMs as actors and judges on ALFWorld, and using JEV as a teacher on ALFWorld and WebShop. Reports GRPO policy gains and releases companion code.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#agents
Community
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.