CommunityJev Models & Reimplementations
open-jev DeBERTa-v3-large
Mithril's open, Jev-shaped typed-decision model on DeBERTa-v3-large: choice up to 255 options, score and noul from one forward pass, 0.854 in-domain with ECE 0.022, trained on public gold labels only.
- Published by
- Community
- Author
- Mithril (Kotoba Labs)
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceSelf-hostedRuns locally
Highlights
- A DeBERTa-v3-large encoder reads a state and any number of typed questions in one pass, with choice up to 255 options, score over 2-10 levels and noul.
- In-domain accuracy 0.854 with Brier 0.213 and ECE 0.022 on 1,500 test states and 3,508 questions.
- Out of distribution it drops to 0.690 on 4,012 never-seen question types, with a 0.847 ± 0.005 to 0.678 ± 0.012 spread across three seeds.
- Trained on public gold labels only — banking77, SST-5 and BoolQ — 18,000 states and 42,000 questions in one 229-second H100 run.
- 512-token context (state cut to 256), 1.8 s for 4 questions on an M1 Max CPU and 518 questions/s at batch 8 on H100.
Watch out
It reads the question only partly, as the 0.854 in-domain versus 0.690 out-of-distribution gap shows; unseen ordered scales are weakest, barely above majority. English only, 512-token context and three public domains, with confidence over-confident by about 0.03 on OOD questions, which the card says should be re-calibrated on your own data.
Questions about this entry
Short answers drawn from the linked project or page.
It reproduces the shape of TypeSafe AI's Jev: one state plus typed questions in, a calibrated probability distribution per question out of a single forward pass, nothing generated, so the structured-output error rate is zero by construction.
No. The card states it is an independent reproduction of the shape, not affiliated with TypeSafe, using none of their data or code, and that its numbers are not comparable because it is measured on public gold labels rather than agreement with frontier models on private workflows.
A single DeBERTa-v3-large pass reads the state and all questions: each option is scored from mean question tokens, mean option tokens and their product, and a softmax within each question's option group forms that question's distribution. Every answer ships with the full distribution.
CPU fp32 on an M1 Max takes 1.8 s for 4 questions; on an H100 in bf16 it is 28 ms end-to-end for 10 questions on one state (25 ms forward) and 518 questions per second at batch 8.
Public gold labels only, no synthetic answers and no teacher model: mteb/banking77 customer messages, SetFit/sst5 review sentences and google/boolq passages, totalling 18,000 train states and 42,000 questions for one epoch, seed 2, in 229 seconds on one H100.
It reads only part of the question, with unseen ordered scales near majority (0.45 against 0.26), English-only 512-token input and three public domains. On out-of-distribution questions its confidence is over-confident by about 0.03 mean, and it cannot generate text or arguments.
Published by Mithril (formerly Kotoba Cloud), operated by Kotoba Labs Inc. The base is microsoft/deberta-v3-large under MIT, datasets are banking77 (CC-BY-4.0), SST-5 and BoolQ (CC-BY-SA-3.0), and code and measurements live at kotoba-lang/typed-decisions.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsHacker News: A one-token logprob wrapper that also classifies webcam frames.
allanrbo151 pointsA single function Jev-like wrapper for LLMs, including vision modelsRead the thread on Hacker NewsMore Jev resources like this
AutoTrust's combined System 1 and System 2 bundle on Gemma-4-26B-A4B-it: Decision Index 0.2.1 of 58.05, a 24-slot head with noul, choice and score, and the base chat model left bit-identical.
Jev Models & Reimplementations#community#open-models#local
Community
akhilaaa3's merged multimodal decision classifier on Gemma 4 12B IT: probabilities for text, image, audio or video options, 87.57% on DecisionBench Medium and MMAU 63.10%, Apache-2.0.
Jev Models & Reimplementations#community#open-models#local
Community
alibiserikbay's Apache-2.0 Jev alternative on Qwen3.5-4B with a merged rank-16 LoRA: one forward pass returns a probability per option, 0.784 on JevBench's public hard tier at about 13 ms per decision.
Jev Models & Reimplementations#community#open-models#local
Community
The 9B JevK5: the same v0.3 data on Qwen3.5-9B, retrained for 1.5 epochs by a pre-registered sweep, reaching index proxy 0.794 and a 0.859 hand-written hard set, Apache-2.0.
Jev Models & Reimplementations#community#open-models#local
Community
The v3 Jev-Style decision model at 2B in GGUF: F16 3.78 GB and Q8_0 2.01 GB match PyTorch FP32 on 100% and 99.7% of parity rows, 73.6% on JevBench v1.4.1 public items.
Jev Models & Reimplementations#community#open-models#local
Community
A CPU-first BitNet student: Microsoft's BitNet b1.58 2B quantized to an I2_S GGUF with a float32 pointer head, distilled from Kev 9B logits and run through the bit-jev Python package with no separate open-weight licence.
Jev Models & Reimplementations#community#open-models#local
Community
From the Jev guides
Original write-ups on what this entry is for.
Training recipes, looped architectures, and language ports: what the open side of the typed-decision class shipped this month, and how to pick by runtime and calibration instead of the leaderboard.
Jev Models & Reimplementations#open-models#training#local
Read article
Taking a decision checkpoint out of Python: what the GGUF and browser WebGPU exports preserve, what they change, and the parity evidence the model cards publish.
Jev Tools & CLIs#open-models#local#onnx
Read article
Interface, size, license, claims — in that order. A selection guide across the system-one catalog, from a 706K form scorer to 27B Jev students.
Jev Practices & Patterns#open-models#evaluation#benchmarks
Read article
From the Jev community
Posts from builders shipping with Jev right now.