Jev directory
CommunityJev Models & Reimplementations
Gemma 3 270M Typed Decisions
Gemma 3 270M fully fine-tuned as a probabilistic decision model: 0.761 accuracy and 0.1319 KL on the 400-case typed-decisions test set, reading letter-token logits one question per pass.
- Reading time
- 2 min
- Published by
- Community
- Author
- rony
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceSelf-hostedRuns locally
Highlights
- 0.761 accuracy, 0.1319 KL, 0.0716 Brier, and 0.1480 per-question ECE on 400 test cases and 2,000 decisions never seen in training.
- Baselines on the same split: prior 0.478 accuracy and zero-shot Gemma 3 270M 0.279, against the fine-tune's 0.761.
- Each (case, question) prompt lists options as letters A, B, C, and the logits at the reply position restricted to those letters are softmaxed.
- 68 ms per case on CUDA; 1,080 cases trained with 120 held out for epoch selection on validation KL.
- Per type: noul 0.822, score 0.739, and choice 0.732 accuracy, with ECE 0.066, 0.158, and 0.112.
Watch out
The gemma licence applies, and the gold labels are the mean of three teacher samples, so scores measure agreement with that teacher, whose self-agreement is about 0.735. ECE is averaged per question (0.1480), not pooled (0.1146), and all figures are the card's own run on the held-out split.
Questions about this entry
Short answers drawn from the linked project or page.
As a full fine-tune of google/gemma-3-270m-it on all four typed-decisions workflows: 1,080 cases for training and 120 held out for epoch selection on validation KL.
On the same test split the prior baseline scores 0.478 accuracy and 0.327 KL, and zero-shot Gemma 3 270M scores 0.279 and 1.068; the fine-tune reaches 0.761 accuracy and 0.1319 KL.
They are the mean of three teacher samples, so the scores measure agreement with that teacher. The card notes the teacher self-agreement is about 0.735, the ceiling any student of it approaches.
Each (case, question) pair is one prompt listing options as letters A, B, C and so on; the next-token logits at the reply position are restricted to those letter tokens and softmaxed, so a case takes five forward passes for its five questions.
choice 0.732 with KL 0.167 and ECE 0.112, noul 0.822 with KL 0.089 and ECE 0.066, and score 0.739 with KL 0.138 and ECE 0.158.
Latency is 68 ms per case on CUDA. ECE is computed per question and averaged over the 20 questions at 0.1480; pooled over all decisions it is 0.1146, so the figure depends on how you aggregate.
All four: agent_trace_observability, customer_service, invoice_processing, and security_incidents, scored by workflow on test at 0.740, 0.772, 0.784, and 0.750 accuracy.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
An EmbeddingGemma 2 fine-tune that answers choice, score, and noul questions from a state and a plain-data schema, with a 0.5M-parameter schema-conditioned scorer over 768-d embeddings.
Jev Models & Reimplementations#community#open-models#local
Community
The fine-tuned Laya checkpoint for four typed-decision workflows: 0.766 accuracy on 2,000 held-out decisions, above Jev 1.13.0's published 0.727 and the teacher self-agreement ceiling it trained past.
Jev Models & Reimplementations#community#open-models#local
Community
A LoRA plus 24-slot decision head over Qwen3.8-27B from Ming Xu's train-jev project: it serves TypeSafe's System One API with 0.851 Kev transfer-v4 accuracy and ECE 0.053, and needs an 80 GB GPU.
Jev Models & Reimplementations#community#open-models#local
Community
A 24 GB-GPU alternative from the train-jev project: Qwen3.5-4B plus a rank-16 LoRA and 24-slot decision head scoring 0.647 PK accuracy, 40.4 Decision Index and 43 ms p50 latency, ahead of Kev-4B at the same size.
Jev Models & Reimplementations#community#open-models#local
Community
A Laya fine-tune for coding decisions: roughly 9k rows from SWE-Gym teach keep/drop file relevance, priority scoring and injection detection, reporting 0.833 accuracy on 1,150 held-out rows at 126 ms p50.
Jev Models & Reimplementations#community#open-models#local
Community
A Laya fine-tune from Convai Innovations on the 1,200-case typed-decisions benchmark: it reports 0.777 accuracy on the official 400-case test set, above TypeSafe Jev 1.13.0 at 0.727 and the teacher ceiling at 0.735.
Jev Models & Reimplementations#community#open-models#local
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.