CommunityJev Models & Reimplementations
ModernBERT-ja 310M Jev
An unofficial cross-encoder fine-tune of ModernBERT-ja-310m for Jev Local: it reads a question-plus-state and one candidate per sequence and hits 92.62% on the JGLUE JNLI test, with CC BY-SA 4.0 weights.
- Published by
- Community
- Author
- argos1111
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open sourceSelf-hostedRuns locally
Highlights
- Cross-encoder over sbintuitions/modernbert-ja-310m at commit 77675fc9 with a single-label head and CLS pooling.
- Listwise cross-entropy over 94,384 questions (~345,000 pairs) built from JGLUE, JCoLA, JCommonsenseMorality and MASSIVE.
- JGLUE test: JNLI 92.62% (2323/2508) and JCommonsenseQA 92.40% (1033/1118) through the local HTTP API.
- Out of distribution: livedoor 35.6% against LFM2.5-1.2B's 46.8% and JMMLU 34.8% against 43.6%.
- Weights are CC BY-SA 4.0, and prompts must keep the training rendering that Jev Local checks as modernbert-jev/1.
Watch out
Unofficial and unrelated to TypeSafe's Jev; it does not reproduce Jev's training or accuracy. Confidence is distribution concentration, not calibrated accuracy, and wrong JMMLU answers averaged a 0.66 maximum probability. No image input, only Japanese and English evaluated, state beyond 512 tokens truncated, and the CC BY-SA 4.0 weights inherit the training datasets' licences.
Questions about this entry
Short answers drawn from the linked project or page.
It is an unofficial cross-encoder for Jev Local's ModernBERT backend and is unrelated to TypeSafe's Jev; the card states it does not reproduce Jev's training or accuracy. It reads a question-plus-state and one candidate as a single sequence and softmaxes one score per candidate.
The rendering must match training, with the question before the state and candidates as label-and-description lines. JSON states are serialized with json.dumps(..., ensure_ascii=False), Noul uses the candidates true and false, and Jev Local checks format_version modernbert-jev/1 from jev_modernbert.json.
On the JGLUE test split it reaches JNLI 92.62% (2323/2508) and JCommonsenseQA 92.40% (1033/1118) through the local HTTP API at four parallel requests. LFM2.5-1.2B zero-shot scored 17.15% and 68.87% through the same API, but that comparison is not like-for-like since this model trained on the train splits.
Those tasks need knowledge or unseen label systems: livedoor 9-category 35.6% against LFM's 46.8% and JMMLU 34.8% against 43.6%. Short Japanese intent and relation judgments stay stronger, and the 16-example customer-support set scored 93.8% but the card calls it indicative only.
One scorer trained on 94,384 questions (~345,000 pairs) converted from JGLUE, JCoLA, JCommonsenseMorality and MASSIVE ja-JP train splits with listwise cross-entropy: 2 epochs, 5,386 steps, lr 3e-5, BF16, 128 pairs per step, about 41 minutes on an AMD Radeon AI PRO R9700.
CC BY-SA 4.0, a conservative choice the card says inherits conditions from the MIT base model and the CC BY-SA 4.0, CC BY 4.0 and MIT training datasets. Keep the dataset attributions when reusing or redistributing the model.
States over 512 tokens are truncated from the end, images are unsupported, and only Japanese and English were evaluated. Answers outside the trained question types (NLI, intent, stage rating, common-sense QA, truth) are not guaranteed, and the model does not follow instructions like an LLM.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsHacker News: A one-token logprob wrapper that also classifies webcam frames.
allanrbo151 pointsA single function Jev-like wrapper for LLMs, including vision modelsRead the thread on Hacker NewsMore Jev resources like this
A 4B decision model taking text, JSON, or images up to 262,144 tokens: 52.75 on Decision Index 0.2.1, 204 of 231 on JevBench public, and 26.0 ms for a three-question request on one H200.
Jev Models & Reimplementations#community#open-models#local
Community
Mithril's open, Jev-shaped typed-decision model on DeBERTa-v3-large: choice up to 255 options, score and noul from one forward pass, 0.854 in-domain with ECE 0.022, trained on public gold labels only.
Jev Models & Reimplementations#community#open-models#local
Community
The 322M mmBERT-base build of Laya for 100+ languages: state plus typed questions return typed answers with probabilities in one forward pass, with up to 8,192 tokens for long documents.
Jev Models & Reimplementations#community#open-models#local
Community
An Open Classification Foundation smoke checkpoint: a 164M-parameter ModernBERT-base with a cross-entropy decision head, 0.751 accuracy and 0.044 ECE on 2,000 held-out typed decisions.
Jev Models & Reimplementations#community#open-models#local
Community
An open Thai + English System One decision model at 0.8B, Apache-2.0: choice with up to 255 options, fractional score, and noul answers in one forward pass over a Qwen3.5-0.8B text tower.
Jev Models & Reimplementations#community#open-models#local
Community
A LoRA scorer on Qwen3.5-4B-Base that scores (state, question, option) triples with a scalar head and softmaxes per question, reaching 0.707 accuracy and 0.044 ECE at T=1.75.
Jev Models & Reimplementations#community#open-models#local
Community
From the Jev guides
Original write-ups on what this entry is for.
Training recipes, looped architectures, and language ports: what the open side of the typed-decision class shipped this month, and how to pick by runtime and calibration instead of the leaderboard.
Jev Models & Reimplementations#open-models#training#local
Read article
Taking a decision checkpoint out of Python: what the GGUF and browser WebGPU exports preserve, what they change, and the parity evidence the model cards publish.
Jev Tools & CLIs#open-models#local#onnx
Read article
Interface, size, license, claims — in that order. A selection guide across the system-one catalog, from a 706K form scorer to 27B Jev students.
Jev Practices & Patterns#open-models#evaluation#benchmarks
Read article
From the Jev community
Posts from builders shipping with Jev right now.