How large is the accuracy gap to the closed model?

On the shared strict subset the replica trails the published model, beats a stock mid-size baseline, and matches closely on demo families while dropping on never-trained task types.

Why was the platformer demo dropped from the lineup?

The base model played one game zero-shot but stalled on the other with mostly no-op actions; a quick fine-tune fixed it but was deemed unfair, so it was left out.

What cold-start behavior should I expect from the live demo?

The demo endpoint scales to zero, so the first request after idle can take about a minute; nothing installs locally beyond the cloud CLI.

How does it handle more options than fit in one pass?

Single-token option letters cap one pass at several dozen choices; larger sets are chunked with a none-of-the-above slot and a final round.

What is the training and serving stack?

It is built on Gemma 4 E2B with an attention LoRA and a smaller Gemma 3 270M tier, trained and served entirely on Modal. Training uses cross-entropy plus Brier loss with temperature scaling, and serving is a one-origin FastAPI app exposing GET /, POST /decide, and GET /info.

What datasets was it trained on?

92 public Hugging Face decision datasets covering intents, routing, moderation, NLI, fact-checking, relevance, QA, rubric scores, pairwise judging, tool selection, and agent next-action, of which 23 are held out and never trained. Rule-based generators add synthetic data for the demo families such as game state, support triage, and invoice packets.

What do the email and latency numbers look like?

On 1,000 real emails with four decisions each it ran 13.4 seconds at 74.6 emails per second with 95.4% spam accuracy, and the 27-question launch demo answered in 97 ms on an H100 against 114 ms for the published model. Demo task families scored 98.8% with an ECE of 0.003, while held-out task types reached 74.8%.

CommunityJev Practices & Patterns

System One Open

Open replica of the typed decision API on openly available base models, answering choice, score, and true-probability questions in one forward pass with no decoding.

See highlights

Open github.com

Stars: 39
Language: Python
Published by: Community
Author: mithalouni
Last verified: Oct 2026

mithalouni/system-one-open

User repository on GitHub

View on GitHub

Open replica of TypeSafe's Jev: typed calibrated decisions in one forward pass, on Gemma 4 E2B / Gemma 3 270M (Modal)

Forks: 6
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

State in, typed calibrated decisions out, one forward pass, no decoding. An open replica of TypeSafe's Jev built on Gemma 4 E2B (attention LoRA) and Gemma 3 270M, trained and served on Modal. MIT licensed. Full report with baselines and caveats: Base URL: (scales to zero; first request after idle takes 60 s).

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

Same three-question shape as the closed API, with distributions and confidence per answer.
Single forward pass with slot-logit scoring; chunked runoff covers oversized option sets.
Eight recorded demos span email triage, games, browser steps, driving, and smart home.
Full pipeline from data mixture and eval rebuild through training and serving is included.

Watch out

The Mario task was dropped after the base model stalled on no-op actions, and weights ship on a cloud volume with public upload still pending. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.