Jev directory
CommunityJev Models & Reimplementations
Ministral 3 3B Reasoning Typed Decisions
A specialist full fine-tune of mistralai/Ministral-3-3B-Reasoning-2512 by rony for the LocalLLaMA/typed-decisions benchmark: it reads a case, question and lettered options and softmaxes next-token logits, reporting 0.808 accuracy and 0.0773 KL on all/test.
- Reading time
- 2 min
- Published by
- Community
- Author
- rony
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceSelf-hostedRuns locally
Highlights
- Full fine-tune of Ministral-3-3B-Reasoning with the vision tower dropped and the input embedding and tied LM head frozen.
- Reports 0.808 accuracy, 0.0773 KL, 0.0424 Brier and 0.1822 per-question ECE on all/test: 400 cases, 2,000 decisions.
- By type: noul 0.853, choice 0.803, score 0.779; by workflow: invoice processing 0.866, customer service 0.836, security incidents 0.784.
- 373 ms per case on an NVIDIA GB10; one forward pass per question, five per case, with no reasoning tokens generated.
- Baselines on the same split: prior accuracy 0.478 and zero-shot Ministral 3 3B Reasoning 0.340.
Quickstart
python
import json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "rony/ministral3-3b-reasoning-typed-decisions"
tok = AutoTokenizer.from_pretrained(repo, fix_mistral_regex=True)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).eval()Watch out
Card reviewed 2026-10-17; all metrics are self-reported and the gold set is the mean of three teacher samples with about 0.735 self-agreement, so 0.808 measures agreement with that teacher rather than ground truth. ECE is computed per question and averaged over 20 questions; pooled over all decisions it is 0.1742.
Questions about this entry
Short answers drawn from the linked project or page.
A specialist model by rony, fine-tuned only for the LocalLLaMA/typed-decisions benchmark; the card says it is not a general-purpose chat or reasoning model. It full fine-tunes Ministral-3-3B-Reasoning as a probabilistic decision model over the dataset's choice, noul and score questions.
Each case-question pair is one prompt: a short system turn, the case JSON, the question and its options listed as letters A, B, C. The assistant turn is pre-filled with an empty THINK block, so the next-token logits at that position, restricted to the letter tokens, give the answer distribution.
On all/test (400 cases, 2,000 decisions): accuracy 0.808, KL 0.0773, Brier 0.0424 and per-question ECE 0.1822. By type noul scores 0.853, choice 0.803 and score 0.779; latency is 373 ms per case on an NVIDIA GB10.
The model was fine-tuned on LocalLLaMA/typed-decisions train, all four workflows, using 1,080 cases with 120 held out for epoch selection on validation KL. Training minimises soft cross-entropy against the gold distribution.
On the same split the card lists prior accuracy 0.478 and zero-shot Ministral 3 3B Reasoning at 0.340, both far below the fine-tune's 0.808. Gold is the mean of three teacher samples, so the scores measure agreement with that teacher, whose self-agreement is about 0.735.
Load it with transformers AutoTokenizer and AutoModelForCausalLM in bfloat16, build the prompt with the lettered options and read the next-token logits. The card links a jupyter notebook, plus test_predictions.jsonl and test_metrics.json for the full test distributions and metrics.
Apache-2.0, fine-tuned from mistralai/Ministral-3-3B-Reasoning-2512. The card frames it as a benchmark specialist: scores measure agreement with a teacher-derived gold rather than ground truth, and behaviour outside the Typed Decisions distribution is not documented.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
A Laya fine-tune from Convai Innovations on the 1,200-case typed-decisions benchmark: it reports 0.777 accuracy on the official 400-case test set, above TypeSafe Jev 1.13.0 at 0.727 and the teacher ceiling at 0.735.
Jev Models & Reimplementations#community#open-models#local
Community
AutoTrust's combined System 1 and System 2 bundle on Gemma-4-26B-A4B-it: Decision Index 0.2.1 of 58.05, a 24-slot head with noul, choice and score, and the base chat model left bit-identical.
Jev Models & Reimplementations#community#open-models#local
Community
The fine-tuned Laya checkpoint for four typed-decision workflows: 0.766 accuracy on 2,000 held-out decisions, above Jev 1.13.0's published 0.727 and the teacher self-agreement ceiling it trained past.
Jev Models & Reimplementations#community#open-models#local
Community
A LoRA plus 24-slot decision head over Qwen3.8-27B from Ming Xu's train-jev project: it serves TypeSafe's System One API with 0.851 Kev transfer-v4 accuracy and ECE 0.053, and needs an 80 GB GPU.
Jev Models & Reimplementations#community#open-models#local
Community
A 24 GB-GPU alternative from the train-jev project: Qwen3.5-4B plus a rank-16 LoRA and 24-slot decision head scoring 0.647 PK accuracy, 40.4 Decision Index and 43 ms p50 latency, ahead of Kev-4B at the same size.
Jev Models & Reimplementations#community#open-models#local
Community
A Laya fine-tune for coding decisions: roughly 9k rows from SWE-Gym teach keep/drop file relevance, priority scoring and injection detection, reporting 0.833 accuracy on 1,150 held-out rows at 126 ms p50.
Jev Models & Reimplementations#community#open-models#local
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.