User repository on GitHub
- Forks
- 0
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
System One on Gemma 4 E4B
A Python experiment that adapts Gemma 4 E4B for System One-style decisions: one call takes a situation and a fixed answer list and returns a probability per answer, with per-dataset adapters and held-out results across four datasets.
- Stars
- 0
- Language
- Python
- Reading time
- 2 min
- Published by
- Community
- Author
- omnaiduu
- Last verified
- Oct 2026
- Languages
- Python
- Capabilities
- Open source
Highlights
- One call takes a situation and a fixed list of answers and returns a probability for each answer.
- Gemma 4 E4B is text only, and each dataset has its own adapter; adapters are not merged into Gemma.
- On typed-decisions, the adapter scores 1,572 of 2,000 versus 1,219 for the plain model on held-out rows.
- Banking77 uses 20 intents per row, not all 77, and improves from 2,547 to 2,958 of 3,076.
- The keep rule is recorded in results/phase6/keep.json; SNLI was not trained on.
Watch out
README reviewed 2026-10-17; the captured writeup shows benchmark counts and code pointers but cuts off inside the core/ description, so training setup, hardware, licence and adapter weights are not covered. Counts are correct answers on rows the adapter did not train on.
Questions about this entry
Short answers drawn from the linked project or page.
A Python experiment that turns Gemma 4 E4B into a System One-style decision model: one call takes a situation and a fixed answer list and returns a probability per answer.
Each dataset has its own adapter, and the adapters are not merged into Gemma, according to the README. The code under judge/ runs the table with one adapter per dataset.
On held-out rows, the adapter improves BoolQ to 2,972/3,270, Banking77 to 2,958/3,076, MultiNLI to 8,984/9,815 and typed-decisions to 1,572/2,000.
The README says Banking77 shows 20 intents on each row rather than all 77, so the reported count reflects the 20-intent setup used in the experiment.
The keep rule is recorded in results/phase6/keep.json. The README notes that SNLI was not trained on and that its MultiNLI adaptation score does not decide whether the adapter is kept.
The README frames the work as System One modelling on Gemma and includes a typed-decisions dataset, but it does not say it calls or distils Jev; the connection is the decision format, not a Jev integration.
The excerpt cuts off inside the core/ description, so training hardware, data licences, adapter weights, install steps and full evaluation methodology are not available in the digest.
More Jev resources like this
A benchmark of Jev and other decision models on trolley-problem states, with runners, raw answers and analysis in one repository. The v1 snapshot covers 1,098 states and the October 2026 v2 run has 14 models on 5,240 frozen states.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
zachlandes/trolley-benchHTML
zachlandes/trolley-benchHTMLA reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonA benchmark harness and web UI that cross-compares dozens of zero-shot information-extraction and classification systems — extractor encoders, rerankers, and typed-decision engines like Jev — on accuracy, latency, and cost.
Jev Benchmarks & Evaluations#community#python#benchmark
Community
umstek/zero-shot-ie-benchPython
umstek/zero-shot-ie-benchPythonAkitaOnRails' study documentation and benchmark harness for Jev-like decision models: frozen inputs, inspectable expected answers, raw response archives and reproducible scoring compare TypeSafe Jev with Decision-1, OpenAI Decisions, Clef, Decision 3.0, Laya, Kev, Eikos, Jebadiah, Mapika Decider and GLiNER.
Jev Benchmarks & Evaluations#community#github#python
Code and data for evaluating Jev, a System One model, on scientific decisions and how its choices affect downstream results. The S2 collection has 20 groups, 40 scientific Choices and one engineering Choice, scored by deterministic downstream code.
Jev Benchmarks & Evaluations#community#github#benchmark
A study of script-dependent answers about Taiwan in commercial decision models: 28 statements asked in Traditional Chinese, Simplified Chinese and English to OpenAI, TypeSafe Jev and Cloudflare models, with data, code and a paper.
Jev Benchmarks & Evaluations#community#github#research
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.