jujumilk3/jev-calibration-audit
User repository on GitHub
Independent API-only calibration audit of TypeSafe AI's Jev decision model
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
Jev Calibration Audit
An unaffiliated, API-only audit of Jev calibration built on roughly 7,000 public HTTP calls, covering abstention, Korean, probabilistic coherence, option order, question interference, caching and latency, with a drift ledger.
- Stars
- 0
- Language
- Python
- License
- MIT
- Published by
- Community
- Author
- jujumilk3
- Last verified
- Oct 2026
About this project
An independent, API-only audit of the calibration of TypeSafe AI's Jev, a "System One" decision model that answers pre-defined questions over a text state with probability distributions instead of generated text. Not affiliated with TypeSafe AI. No model weights, no fine-tuning, no distillation — every number here comes from public HTTP calls and public datasets.
Quoted from the project's README. Read the full README.
Works forJev for Classification
Used inJev for Software & DevTools
- Languages
- Python
Highlights
- Remove the unknown option and accuracy on unanswerable items drops from 0.950 to 0.000 while the stereotype rate rises from 0.03 to 0.79.
- Option-order bias came back as none: mean shift 0.005, zero argmax flips in 400 trials, and 16 bundled questions shifted confidence 0.008.
- The probability is not a property of the judgement: complements summed to 0.71-1.42, and Noul vs Choice differed by 0.125.
- Method notes are as load-bearing as the numbers: report ECE against a noise floor rather than zero, since a perfect model still scores about 0.045 at n=60.
Watch out
This is one independent researcher publishing against a model he does not control, with no affiliation to TypeSafe, and every figure comes from public HTTP calls rather than weights. The README's opening complaint is precisely that published measurements disagree and that none report confidence intervals, so read the bootstrap CIs in FINDINGS.md alongside the headline numbers. Model aliases move, which is why every response's model field is logged and why the drift ledger has to be run on a schedule to be useful at all, and why the confidence field is undocumented and all metrics are computed from probabilities instead. The state-blind control is the part most benchmarks skip: shown the options with the question removed, the model still scores 0.38-0.46 against a chance rate near 0.15. Datasets are not redistributed here, only results. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
They come from about 7,000 live API calls at under a US dollar in total, with raw per-call records committed under results/raw/ and the code MIT licensed. No weights, fine-tuning or distillation are involved, so anyone can rerun the same probes. Treat it as one careful independent measurement rather than a settled verdict.
Removing the abstain option. Accuracy on unanswerable items collapsed from 0.950 to 0.000, the stereotype rate rose from 0.03 to 0.79 at 0.79 confidence, and ECE went from 0.023 to 0.793. If your use case has genuinely unanswerable inputs, always keep an explicit unknown option in the choice list.
Not safely. The method notes say ECE is not comparable across models at different accuracies, that bin-based ECE needs bootstrap CIs, and that at n=60 a perfect model still scores about 0.045, so small-n numbers are noise in either direction. Always report against that noise floor rather than against zero.
That is what the drift ledger is for. The stated policy is that a published benchmark retires the moment it is published and the version aliases move, so a fixed bilingual probe set run daily is the only way to see the change happen, and it cannot be backfilled later. Ten probes a day costs well under a dollar a month, and each response's model field is logged for the same reason.
Seven experiments and about 7,000 API calls for under one US dollar. Every number comes from public HTTP calls and public datasets, with no model weights, fine-tuning or distillation.
At n=60 even a perfect model still scores about 0.045 ECE, so the README says to report ECE against a noise floor rather than against zero, or small-n claims in either direction are noise.
On unanswerable items accuracy fell from 0.950 to 0.000, the stereotype rate rose from 0.03 to 0.79, and ECE went from 0.023 to 0.793.
More Jev resources like this
24GitHub stars
Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rorshopping/jev-on-a-laptopPython
rorshopping/jev-on-a-laptopPython22GitHub stars
Offline evaluator that checks whether classifier confidences match reality and converts mistake costs into a deployable threshold, emitting an HTML report plus machine-readable threshold files.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
rlaope/jevalPython
rlaope/jevalPython16GitHub stars
Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPython37GitHub stars
Minimal Python recipe that turns any causal LLM into a Jev-style classifier by prefilling choice_index and constraining sampling to legal indexes in one forward pass.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
sgoedecke/system-onePython
sgoedecke/system-onePython39GitHub stars
Open replica of the typed decision API on openly available base models, answering choice, score, and true-probability questions in one forward pass with no decoding.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
mithalouni/system-one-openPython
mithalouni/system-one-openPython941GitHub stars
Type-safe generation layer over existing autoregressive models via JSON Schema on SGLang, returning constrained string, numeric, boolean, and enum answers with optional thinking.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
zmtomorrow/TypeARPython
zmtomorrow/TypeARPythonFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.