CommunityJev Models & Reimplementations
System One Scorer
A LoRA scorer on Qwen3.5-4B-Base that scores (state, question, option) triples with a scalar head and softmaxes per question, reaching 0.707 accuracy and 0.044 ECE at T=1.75.
- Published by
- Community
- Author
- pngwn
- Last verified
- Oct 2026
- Capabilities
- Open sourceSelf-hostedRuns locally
Highlights
- 30.5M trainable of 4.24B: LoRA r=16 over all linear projections plus a new scalar score head, under a CC-BY-NC-4.0 licence.
- Test: 0.707 accuracy, 0.044 ECE, and 0.373 Brier across 576 questions after fitting T=1.75 on validation.
- Raw-head ECE was 0.135; temperature scaling cuts it to 0.044 at identical accuracy.
- 112.3 ms per question at 4 options; latency scales with option count and sequence length, not output length.
- Validation across all nine families with a per-task cap of 64 scored 0.752 accuracy and 0.032 ECE.
Watch out
The ticket component of the training data is CC-BY-NC-4.0, so the model inherits a non-commercial restriction. High-cardinality tasks were trained with a 16-option cap but evaluated uncapped (tickets_queue reaches 0.234 accuracy), 384-token truncation weakens MMLU and yelp_score, and the card says accuracy is far below frontier models.
Questions about this entry
Short answers drawn from the linked project or page.
Neither. It scores (state, question, option) triples with a sequence-classification head and softmaxes the per-question logits; it cannot produce free text, and the output space is the option set by construction.
The raw head is over-confident: ECE 0.135 on test. Fitting a temperature of 1.75 on validation cuts ECE to 0.044 at identical 0.707 accuracy, about twice as well calibrated as the prompted baseline.
The prompted Qwen3.5-4B-Base 26-letter baseline scores 0.679 accuracy and 0.093 ECE on 112 validation questions; the comparison covers 7 of 9 families because banking77 and ticket routing exceed its answer alphabet.
Option-cardinality mismatch: high-cardinality tasks are trained with a 16-option cap for batching but evaluated over all options (52 for ticket routing), and tickets_queue reaches only 0.234 accuracy with 0.244 ECE.
CC-BY-NC-4.0, inherited because the ticket component of the training data is CC-BY-NC-4.0, which makes the model non-commercial for that reason.
On pngwn/system-one-decisions, 12,913 train questions across 9 families: 2,200 optimizer steps, batch 8 questions, max_len 384, option cap 16, and lr 1e-4 cosine with 3% warmup on an a100-large in 2h58m.
At 4 options one question takes 112.3 ms; a smoke run at 77 options took 559.7 ms. Latency scales with option count and sequence length rather than output length, because nothing is generated.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsHacker News: A one-token logprob wrapper that also classifies webcam frames.
allanrbo151 pointsA single function Jev-like wrapper for LLMs, including vision modelsRead the thread on Hacker NewsRelated terms
Glossary definitions related to this entry.
More Jev resources like this
An unofficial cross-encoder fine-tune of ModernBERT-ja-310m for Jev Local: it reads a question-plus-state and one candidate per sequence and hits 92.62% on the JGLUE JNLI test, with CC BY-SA 4.0 weights.
Jev Models & Reimplementations#community#open-models#local
Community
An Open Classification Foundation smoke checkpoint: a 164M-parameter ModernBERT-base with a cross-entropy decision head, 0.751 accuracy and 0.044 ECE on 2,000 held-out typed decisions.
Jev Models & Reimplementations#community#open-models#local
Community
A 4B decision model taking text, JSON, or images up to 262,144 tokens: 52.75 on Decision Index 0.2.1, 204 of 231 on JevBench public, and 26.0 ms for a three-question request on one H200.
Jev Models & Reimplementations#community#open-models#local
Community
AutoTrust's combined System 1 and System 2 bundle on Gemma-4-26B-A4B-it: Decision Index 0.2.1 of 58.05, a 24-slot head with noul, choice and score, and the base chat model left bit-identical.
Jev Models & Reimplementations#community#open-models#local
Community
Mithril's open, Jev-shaped typed-decision model on DeBERTa-v3-large: choice up to 255 options, score and noul from one forward pass, 0.854 in-domain with ECE 0.022, trained on public gold labels only.
Jev Models & Reimplementations#community#open-models#local
Community
chenz53's standalone decision model derived from Google Gemma 4 26B A4B IT: full merged BF16 weights, a fitted temperature of 1.736507, and 79.50% equal-type accuracy over 1,100 final questions.
Jev Models & Reimplementations#community#open-models#local
Community
From the Jev guides
Original write-ups on what this entry is for.
Training recipes, looped architectures, and language ports: what the open side of the typed-decision class shipped this month, and how to pick by runtime and calibration instead of the leaderboard.
Jev Models & Reimplementations#open-models#training#local
Read article
Taking a decision checkpoint out of Python: what the GGUF and browser WebGPU exports preserve, what they change, and the parity evidence the model cards publish.
Jev Tools & CLIs#open-models#local#onnx
Read article
Interface, size, license, claims — in that order. A selection guide across the system-one catalog, from a 706K form scorer to 27B Jev students.
Jev Practices & Patterns#open-models#evaluation#benchmarks
Read article
From the Jev community
Posts from builders shipping with Jev right now.