Is 64.4% accuracy actually usable, or just a demo number?

It is held-out accuracy over 12,913 questions across 6 public datasets, and the README itself calls the accuracy modest. The argument is that a calibration error of 0.047 means a stated 80% confidence is roughly right that often, so you can set thresholds, escalate low-confidence cases and build pipelines on it.

How is this different from just calling the real Jev API?

It is an independent implementation of the same scoring shape, trained with LoRA on Gemma 3 270M, not a wrapper around the hosted model. Closed-source Jev is the stated motivation for building it. Interoperability with a hosted Jev endpoint is not something the repository claims.

What does it take to run this on my machine?

Clone, pip install -r requirements.txt, then run infer.py with --state, --question and --options, or app.py for a Gradio UI. Gemma is a gated model, so accept the licence on its model page and log in with huggingface-cli first. CPU inference is fine, and a Colab notebook trains from scratch.

Do the 40-200x faster and 150x cheaper claims survive contact?

They come from the README's own comparison table: roughly 50 ms per decision against 2-10 seconds for GPT-4o or Claude, and about $0.001 per 1,000 decisions against $0.15 per 1,000 calls. It is a static table rather than a reproducible benchmark, and larger Gemma variants are offered for better accuracy without measured results.

How do I run system-one-gemma?

Clone the repo and pip install -r requirements.txt, then train on a free Colab T4 in about 15 minutes or run infer.py, app.py (Gradio), or the Python API locally; CPU works. Gemma is gated, so accept its license and run huggingface-cli login.

What is the base model and how is it packaged?

It is built on Google's Gemma 3 270M, a gated model, and the repo ships a pre-trained LoRA adapter under pretrained-scorer/. A larger Gemma model can be substituted with --base-model for better accuracy on the same architecture.

How can I train it on my own data?

Format data as a list of dicts with task, question_type, state, question, options, and answer_index, then run python system_one.py train --local-data-dir ./my-data --out-dir ./my-model. The Colab notebook trains on six public datasets.

CommunityJev Practices & Patterns

System One Gemma

Open-source System One scorer built on Gemma 3 270M with LoRA, where each option is scored in one batched forward pass and softmaxed into probabilities for choice, yes/no and ordered-scale questions.

See highlights

Open github.com

Stars: 7
Language: Python
Published by: Community
Author: akash-kamat
Last verified: Oct 2026

akash-kamat/system-one-gemma

User repository on GitHub

View on GitHub

Open-source Jev-style System One decision model. Gemma 3 270M with a scoring head — fast, calibrated decisions in a single forward pass. No text generation. Inspired by TypeSafe.ai's Jev.

Forks: 2
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

An open-source implementation of the System One scoring architecture, inspired by TypeSafe.ai's Jev model — the AI that famously plays Doom in real-time. Jev is a massive breakthrough in AI — a model designed for machines, not humans. It doesn't generate text. It makes fast, calibrated decisions from a fixed set of options in a single forward pass. The problem? Jev is closed-source.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

One batched forward pass scores every option; a linear head reads the last token and softmax gives probabilities.
Held-out results over 12,913 questions across 6 tasks: 64.4% accuracy, 0.047 calibration error, 0.454 Brier score.
Runs on CPU locally, or trains in roughly 15 minutes on a free Colab T4 through the bundled notebook.
LoRA rank 16 trains about 2.6M of 268M parameters; last-token pooling keeps option text from being truncated.

Watch out

This reimplements the System One idea rather than shipping Jev, which is closed-source, and 64.4% accuracy is modest even with calibration error at 0.047. Gemma is gated, so the licence must be accepted and you must log in before weights download. The 40-200x latency and 150x cost comparison is a static table against GPT-4o and Claude, not a benchmark you can rerun. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.