This accuracy table looks like it is competing with Jev's published numbers. Is it?

No, and the README says so directly: Jev's 67.8% comes from four private workflows and is not reproducible from outside, so these public-gold numbers are not comparable to that column. What is comparable is the ranking inside this repository — DeBERTa-v3-large at 0.855, LLaDA-MoE at 0.835 and ModernBERT-base at 0.717 on the same 3,508 test questions.

Is a 7B diffusion model worth it when a 435M encoder is more accurate?

Not at this scale. On equal data the diffusion model's edge over the best encoder is 1.6 points, while its latency is 16x worse — 676 ms against 42 ms end to end for one state with ten questions — and training costs $2.29 against $0.26. Extra diffusion steps add 0.3 points for double the latency, so a single pass is the stated operating point.

How much of this result depends on ModernBERT-large underperforming?

Enough that the README calls it a measurement that overturned a prior. ModernBERT-large sat at 0.39 while the base model learned immediately with the span-pooling head, and pushing the base to 6 epochs reached 0.746 while memorising the two short sources and degrading ECE to 0.12. The recommendation is therefore DeBERTa-v3-large, not ModernBERT-large, which was the original candidate.

What did this not measure?

The README has a section on it. Out-of-distribution questions — unseen instructions and option lists against a familiar state — which is the check for whether the model reads the question or memorised the slot. Also distillation from a frontier teacher, which is the target behind Jev's number, multiple seeds, ModernBERT-large beyond a single epoch, and Apple M1 Max latency, where the session died twice.

Which backbones were compared?

ModernBERT-base and -large encoders against inclusionAI/LLaDA-MoE-7B-A1B, a dLLM. The encoder is full fine-tuned and the dLLM uses LoRA r16, with the same CE plus Brier loss and the same latency bench.

Where do the labels come from?

Only public dataset gold labels, from mteb/banking77, SetFit/sst5 and google/boolq. There are no synthetic answers and no teacher model, so the accuracies are not comparable to Jev's own published number.

How were the runs executed?

In a Modal volume on an H100 80GB: uv venv with Python 3.12, modal run modal_app.py::encoder for the encoder and ::dllm for the dLLM, each writing a report.json with loss curve, metrics, latency and USD.

CommunityJev Practices & Patterns

Typed Decisions

A research reproduction of the Jev question shape — Choice, Score and Noul answered in one forward pass — trained and measured side by side on encoder backbones and a diffusion LLM, with accuracy, calibration, latency and training cost in one table.

See highlights

Open github.com

Stars: 10
Language: Python
Published by: Community
Author: kotoba-lang
Last verified: Oct 2026

kotoba-lang/typed-decisions

Organization repository on GitHub

View on GitHub

Jev-shaped typed-decision model (state + Choice/Score/Noul questions -> calibrated probabilities, one pass) on ModernBERT / DeBERTa / LLaDA-MoE, with measured latency, accuracy, calibration and training cost

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

A lightweight reproduction of the Jev shape — one program state, many typed questions (Choice / Score / Noul), calibrated probability distributions back in a single parallel pass — on two backbones, so their speed, accuracy/calibration and training cost can be read side by side. Subject-plane name (kotoba-lang/typed-decisions): the subject is the typed-decision model, not a role and not an origin — Jev (typesafe.ai) is the reference point, not a spec we implement (there is no public spec; the API shape is…

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Accuracy on 3,508 test questions: DeBERTa-v3-large 0.855, LLaDA-MoE-7B-A1B with LoRA 0.835, ModernBERT-base 0.717, each with Brier and ECE reported alongside.
The 7B diffusion model matches the best encoder on accuracy but is 16x slower per state and roughly 14x the training cost, so one pass is the operating point.
A published ablation: a scoring head on a new [OPT] marker token never leaves the label prior, while reading the mean of each option's text tokens works.
Labels are public gold only — banking77, sst5 and boolq — so these accuracies are not comparable to Jev's own figure on private workflows.

Watch out

This is an independent reimplementation, and the README says so: there is no public Jev spec, so the API shape here was reconstructed from the vendor's site and a published guide. The headline accuracies come from public gold labels, which the README states are not comparable to Jev's own number on four private workflows. The listed gaps are equally concrete. Out-of-distribution question shapes are untested, so it is not shown whether the models read an instruction or memorised the option slot; there are no multiple seeds; ModernBERT-large was not trained to convergence; and Apple MPS latency was never captured. DeBERTa-v3-large wins on accuracy but has a 512-token context with the state cut to 256 tokens, and it cannot hold 100 questions at all. The LLaDA row trains on 6,000 states against 18,000 for the encoders, so the equal-data control is the fairer comparison, and the steps=4 result comes from a 60-question smoke run. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

rorshopping/jev-on-a-laptopPython

jeval

Offline evaluator that checks whether classifier confidences match reality and converts mistake costs into a deployable threshold, emitting an HTML report plus machine-readable threshold files.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

rlaope/jevalPython

jev-rag-benchmark

Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

erendikmenn/jev-rag-benchmarkPython

system-one

Minimal Python recipe that turns any causal LLM into a Jev-style classifier by prefilling choice_index and constraining sampling to legal indexes in one forward pass.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

sgoedecke/system-onePython

system-one-open

Open replica of the typed decision API on openly available base models, answering choice, score, and true-probability questions in one forward pass with no decoding.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

mithalouni/system-one-openPython

TypeAR

941

Type-safe generation layer over existing autoregressive models via JSON Schema on SGLang, returning constrained string, numeric, boolean, and enum answers with optional thinking.

Jev Practices & Patterns#community#jevbest#catalog-import

Community

zmtomorrow/TypeARPython

From the Jev guides

Original write-ups on what this entry is for.

Designing the State: What to Include and What to Drop

Oct 12, 2026

Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.

Jev Practices & Patterns#state#design#primitives

Read article

Noul, End to End: One Probability and a Dozen Thresholds

Oct 12, 2026

A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.

Jev Practices & Patterns#noul#primitives#confidence

Read article

Scoring Without Measuring: Bands, Rubrics, and Composite Triage

Oct 12, 2026

A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.

Jev Practices & Patterns#score#primitives#scoring

Read article

Back to all resources

From the Jev community

Posts from builders shipping with Jev right now.

Follow @typesafeai