User repository on GitHub
First independent head-to-head benchmark of System One decision models (Laya vs Jev) on byte-identical inputs
- Forks
- 2
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
CommunityJev Practices & Patterns
Sysone Bench
Independent harness grading System One decision models on the same sealed manifest and seed with byte-identical inputs, pitting closed Jev against open Laya and constrained Qwen.
- Stars
- 6
- Language
- Python
- License
- MIT
- Published by
- Community
- Author
- instax-dutta
- Last verified
- Oct 2026
About this project
An independent head-to-head benchmark of System One decision models. Laya (open weights), Jev (closed TypeSafe API) and Qwen with parallel constrained decoding (PCD) answer the same 1,190 cases and 1,550 typed questions in one run, from one sealed manifest, at one seed. Inputs are verified byte identical before any number is compared, so the models are graded on the same bytes.
Quoted from the project's README. Read the full README.
- Languages
- Python
Highlights
- Same cases, question dicts, order, and seed for Laya, Jev, and parallel-constrained Qwen from one sealed manifest.
- Run artifacts are append-only with pinned identities, checksum verification, and separate raw versus calibrated error.
- Coverage script derives measured counts from verified run directories instead of hand-kept tallies.
- Report tooling regenerates comparisons and figures only from validated rows and refuses existing output dirs.
Watch out
Ground truth is one reviewer's correction of an AI draft with no second reviewer or agreement metric, so per-language and edge numbers are judgment calls. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
Inputs are verified byte identical before any number is compared, with one sealed manifest, one seed, pinned model identities, and checksum checks inside the container. Two numbers from different prompts are never presented as a comparison.
The sealed three-model release puts the closed Jev reference ahead, with the best open model trailing by a small gap and the top open rows effectively tied at this sample size. The expanded panel separately tracks 43 distinct open-weights measurements.
One human reviewer corrected an AI draft of every answer under a recorded protocol, with no second reviewer and no adjudication. The multilingual suite has no native-speaker second label, so read it as one reviewer's judgment.
Open models run CPU-only in a container with fixed CPU and memory, the Jev key arrives on stdin and lives only in the child environment, and report and graphics tools refuse to emit unless recomputed counts reproduce each sealed summary.
compare.py gives a pairwise comparison with bootstrap and tests, while benchmark.report assembles the report from finished run directories and refuses to emit unless its recomputed counts reproduce each run's sealed summary. benchmark.graphics reads only the validated figure-source.json, so a chart cannot disagree with the rows.
Jev 1.13.0 at 0.9065, Laya 0.3.11 at 0.6863 and Qwen2.5-1.5B with parallel constrained decoding at 0.6048, all on 952 cases and 1,240 scored decisions drawn from one sealed manifest at one seed.
Only tev1-08b was measured on two hosts, giving the one direct read on precision cost: 0.7629 on CPU fp32 against 0.7734 on T4 fp16, a 0.0105 gap on byte-identical inputs at one pinned revision.
More Jev resources like this
24GitHub stars
Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rorshopping/jev-on-a-laptopPython
rorshopping/jev-on-a-laptopPython16GitHub stars
Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPython7GitHub stars
Benchmark harness that tests whether models can forecast federal motion-to-dismiss outcomes from the written record, scored per claim-defendant pair with micro-Brier and clustered intervals.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
johnhughes3/LegalForecastBenchPython
johnhughes3/LegalForecastBenchPython6GitHub stars
Frozen 100-question-per-cell check of TypeSafe Jev on Korean versus English reading, paraphrase, and medical exam items, with recorded responses and uncertainty intervals.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
mahlernim/jev-korean-benchmarkPython
mahlernim/jev-korean-benchmarkPython8GitHub stars
Reproducible harness comparing Jev, Laya, two LLMs, and two BERT variants on SST-2, AG News, and Banking77 for accuracy, calibration, latency, throughput, and cost.
Jev Practices & Patterns#community#awesomejev#catalog-import
Community
dhruvmehra/jevbenchPython
dhruvmehra/jevbenchPython7GitHub stars
Python client and decision toolkit for the Jev System One endpoint with noul, choice, and score primitives, batched asks, composition helpers, concurrent evaluation, and an MCP server.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
docxology/daf-jevPython
docxology/daf-jevPythonFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.