Post
CommunityJev Guides & Articles
Build your own decision model
Nish Tahir's tutorial on how decision models like Jev work: mask an LLM's vocabulary to the option tokens, read the softmax as a probability, then measure calibration and add temperature scaling, using Qwen3-1.7B.
- Format
- Post
- Reading time
- 2 min
- Published by
- Community
- Author
- Nish Tahir
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- Explains the single-pass trick: mask the vocabulary to the option tokens, pick the highest-probability one, and read the softmax as its score.
- Provides a Python script that loads Qwen/Qwen3-1.7B, constrains decoding to options A to E, and prints the prediction with probabilities.
- Reports 725/1221 = 0.5938 accuracy and 0.5844 macro F1 on a CommonsenseQA holdout, rising to 0.6241 accuracy after a fine-tune.
- Shows the calibration gap: in the 0.90-1.00 confidence bin the model was right only 70.09% of the time, motivating temperature scaling.
Watch out
Grounded in the truncated post capture from 2026-10-15; the temperature-scaling section is cut off mid-sentence, and the CommonsenseQA figures are the author's own single run, used to illustrate calibration rather than to benchmark Jev or any other model.
Questions about this entry
Short answers drawn from the linked project or page.
Nish Tahir explains what decision models like Jev do, return calibrated probabilities over fixed options in a single pass, and shows how to emulate the behaviour by constraining a small language model's output tokens.
A normal model generates one token per pass, and the post's example needs 11 passes for its output. A decision model masks the vocabulary to the allowed options, so one pass returns the answer and its probabilities.
Qwen/Qwen3-1.7B, loaded with transformers. The script builds a prompt from a JSON question, restricts decoding to options A through E, and prints the prediction with a probability for every option.
On a random holdout it scored 725/1221 = 0.5938 accuracy with macro F1 0.5844, and a quick fine-tune on the dataset lifted accuracy to 762/1221 = 0.6241, which the post calls not bad for a 1.7B model.
The post bins confidence against accuracy: in the 0.90 to 1.00 bin the model was correct only 70% of the time and in the 0.80 to 0.90 bin about 40%, so its probabilities were overconfident and needed temperature scaling.
No. The post warns that without additional training, the option-token probabilities reflect the model's confidence in the next token, not the chance the answer is correct, which is exactly the gap calibration is meant to close.
Yes, it is Python using torch and transformers, taking an --input JSON file with a question and options A to E. The digest capture truncated the temperature-scaling fix that follows the evaluation section.
More Jev resources like this
The companion source code for the book Mastering TypeSafe Jev: 13 chapters plus two appendices build two capstones on one shared decision client — an autonomous incident and FinTech triage engine on hosted Jev, and a SmartPlant industrial-IoT engine on self-hosted open-source Kev.
Jev Guides & Articles#community#catalog-import#yibie
Chinese-language Jev cookbook with runnable Jupyter notebooks covering the three primitives, 18 official recipes, evaluation method, voice smart-home demo, and local Laya fine-tuning.
Jev Guides & Articles#community#jevbest#catalog-import
Independent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonA curated list tracking the open side of the System One ecosystem — runnable Jev-like models, independent benchmarks, and calibration and constrained-decoding tooling — as the counterpoint to lists about the closed hosted Jev.
Jev Guides & Articles#community#catalog-import#cobanov
A one-hour, six-step workshop built around a fighting game that teaches TypeSafe's Discriminative (Jev) decision model and puts it next to Gemini in a Google ADK workflow. It runs in Cloud Shell or anywhere signed into gcloud, with hosted Jev, a local DiffusionGemma VM, or a rehearsal stand-in.
Jev Guides & Articles#community#github#workshop
Benchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.