User repository on GitHub
Semantic ifs from open models, on a 3090 at home. Independent; not affiliated with Jev or TypeSafe.
- Forks
- 357
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
Jev directory
CommunityJev Practices & Patterns
SemIf
An independent baseline that reads typed option probabilities straight from a frozen Qwen3.5-4B's logits in one forward pass, reproducing Jev's interface pattern with open models rather than Jev's undisclosed model or training.
- Stars
- 4.8k
- Language
- Python
- License
- MIT
- Reading time
- 3 min
- Published by
- Community
- Author
- TheoLeeCJ
- Last verified
- Sep 2026
About this project
Semantic ifs from open models, on a 3090 at home. Independent project; not affiliated with Jev or TypeSafe. Wow! No waitlist. Run it in your browser today. Same frozen 4B model · same state · same 21 questions · measured separately, aligned at t=0 in the replay. Most agent decisions are small: route this, retry that, does the evidence support X? A chat model can answer them, but it spends time generating text that software immediately parses back into an if statement.
Quoted from the project's README. Read the full README.
Works forJev for Evaluation
Used inJev for Software & DevTools
Highlights
- On one RTX 3090, direct typed logits answered 21 binary criteria in 1.023 s versus 5.332 s and 111 output tokens for an autoregressive JSON array.
- Backends cover CUDA Torch, CPU llama.cpp from GGUF, and MLX or PyTorch MPS on Apple Silicon, with prompt hashes matching row for row.
- A WebGPU browser demo runs MiniCPM5 2B and Qwen3.5 4B locally with no waitlist or install.
- Shared mode prefills one long state once and branches it across many criteria in parallel.
- The fixture, exact runner commands, row-level outputs, model revisions, and known failures are committed for audit.
Quickstart
bash
python -m venv .venv
. .venv/bin/activate
pip install -e '.[test]'
CUDA_VISIBLE_DEVICES=0 semif-score --mode direct --model Qwen/Qwen3.5-4B --input examples/decisions.jsonl --output results.jsonlWatch out
MIT-licensed. Not affiliated with TypeSafe; needs Python 3.10+ and a CUDA GPU holding a 4B BF16 model (or a local GGUF for the CPU backend), weights are not included, and scores depend on the quantized weights.
Questions about this entry
Short answers drawn from the linked project or page.
Mostly the author's own fixtures, and the README says so. 0.813 balanced accuracy comes from 144 rows the author wrote; WANLI at 0.637 uses 256 public rows. Jev's 0.883 is read from TypeSafe's published records, not a live call, and covers 102 alignable rows rather than their 711-row aggregate.
The README calls it a systems comparison, not a claim of semantic equivalence. Direct typed logits took 1.023 s and 0 output tokens; the compact autoregressive array took 5.332 s and 111 tokens, and its choices matched direct argmax on only 18 of 21 criteria.
They can be calibrated: per-workload temperature scaling drops out-of-fold ECE from 0.068 to 0.038 on authored decisions and 0.208 to 0.069 on WANLI. Calibration never changes which option is selected, and on two of the three workloads the confidence intervals overlap.
Yes, through the llama.cpp backend with a local GGUF. Prompt construction stays on the pinned reference tokenizer, so prompt_sha256 still matches the Torch backend row for row, but llama.cpp evaluation paths introduce small numerical differences: compare decisions or probabilities with a tolerance, not raw logits bit for bit.
Apple Silicon can use the native MLX backend or PyTorch/MPS; CUDA with a GPU that holds a 4B BF16 model uses the standard scorer; and CPU-only runs the llama.cpp backend from a local GGUF checkpoint with no CUDA device, adding --backend llamacpp and --gguf plus optional --llama-threads. Prompt construction stays on the pinned reference tokenizer across backends.
Direct scores each row independently. If every row shares the exact same state, --mode shared prefills it once and evaluates the criteria in parallel; serial prefix reuse and parallel suffixes go further. On the owned 37-state by 21-criterion workload fresh direct scoring managed 2.33 decisions per second, serial reuse 10.75, and parallel suffixes 20.03.
No. SemIf includes per-workload temperature scaling fitted on labeled decisions, which lowers expected calibration error, for example WANLI from 0.208 raw to 0.069 out of fold, but it does not change the selected option. The improvement is clear on WANLI while the intervals overlap on the authored and Every workloads.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Standard One SH (StandardThinking)
LinkedIn: What multimodal decision models actually change, and what they do not.
LinkedInJevDirectory.org
Hacker News: Decision Index 0.2.1 scores open reproductions of Jev on a frozen suite.
jackbravo4 pointsDecision Index 0.2.1 – 70 open reproductions of TypeSafe Jev's Decision ModelRead the thread on Hacker NewsHacker News: Pointer to a DiffusionGemma plus vLLM tutorial for a Jev-shaped classifier.
1 pointHow to build a Jev-style classifier with DiffusionGemma and vLLMRead the thread on Hacker NewsMore Jev resources like this
Apache-licensed, locally runnable Jev-style decision models (0.8B, 4B, 9B on Qwen3.5) with released weights, training code, a System One-compatible server, frozen eval suites, and a playground.
Jev Practices & Patterns#community#python#open-models
Community
jaredpalmer/kevPython
jaredpalmer/kevPythonAn independent open reproduction of the System One model class: Qwen3.5-based 2B and 35B mixture-of-experts models that return typed Choice, Score, and Noul probabilities in one forward pass, with nothing distilled from Jev.
Jev Practices & Patterns#community#python#open-models
Community
Mapika/deciderPython
Mapika/deciderPythonAn open re-creation of Jev's decision model on an open-weights Qwen3.5-4B: it answers a list of typed questions in one forward pass over POST /v1/systemone in about 200 ms, never writing free text.
Jev Practices & Patterns#community#python#open-models
Community
kshetrajna12/reflexPython
kshetrajna12/reflexPythonAn encoder-side reproduction of Jev built on a retrained GLiClass ModernBERT base (151M): it evaluates multiple typed questions in one non-autoregressive forward pass, returning choices, ordinal scores, and probabilities under 35 ms.
Jev Practices & Patterns#community#python#open-models
Community
Heman10x-NGU/Verdict-open-jevPython
Heman10x-NGU/Verdict-open-jevPythonA from-scratch one-pass option scorer that maps a context and a changing list of text options to one probability per option, adding a hashed n-gram encoder, rival-aware attention, a gated head, and temperature scaling; JSONL data is drop-in compatible with jevlike.
Jev Practices & Patterns#community#python#calibration
Community
olanotolu/jevbetterPython
olanotolu/jevbetterPythonA ModernBERT-based open decision model that evaluates Choice, Score, and Noul schemas in one non-autoregressive pass; Verdict 2.0 reports 77.10% accuracy on 2,000 held-out enterprise decisions.
Jev Practices & Patterns#community#python#open-models
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.