User repository on GitHub
Jev on Indonesia's SNBT 2025 university entrance test: 159 questions, seven subtests, audited answer keys.
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
Snbt Jev Bench
Jev run against a community reconstruction of Indonesia's SNBT 2025 entrance exam: 159 remembered questions across seven subtests, 156 scored, with every request payload and response kept.
- Stars
- 1
- Language
- Python
- License
- MIT
- Published by
- Community
- Author
- misaalya
- Last verified
- Oct 2026
About this project
A benchmark of TypeSafe Jev on SNBT 2025, Indonesia's state university entrance test. The dataset has 159 questions across all seven subtests. Each question includes its own reading passage and figure transcriptions. Results are in RESULTS.md, and raw responses are in data/results/. The questions come from Modul MMA SNBT 2025 by Tim Mangkuk Mi Ayam, a 764-page collection of recalled questions (dataset/MMASNBT2025.pdf, not included here).
Quoted from the project's README. Read the full README.
Used inJev for Software & DevTools
- Languages
- Python
Highlights
- Official papers are never released, so the 159 questions were rebuilt from memory; wording may differ and some are flawed.
- Keys cover only PU, PK and PM and come from the module's authors, who say the answers are not certain; 89 items use single-model labels.
- Scored 47.8% against the module key, 85.4% against the Claude Opus 5 labels, and 69.2% combined across 156 items.
- build_requests.py writes every payload to disk before sending, so the request behind any score can be checked afterwards.
Watch out
Read the two scores as different measurements, not a range: 47.8% is on the 67 items with a key from the module's own authors, who write that the answers are not certain, while 85.4% is on the 89 items with no key at all, labelled by Claude Opus 5 alone, which the README says may be wrong. Nothing here measures Jev against the real exam. Three fill-in questions were excluded because inventing options would test the options rather than the model, and two yes/no tables became three Noul judgments each because guessing there is 50% rather than 20%, leaving 156 of 159 items scored. Ten figures are text transcriptions made by Claude and hand-checked, and one was corrected that way when two lines described as parallel were measured as not parallel. Score accuracy by subtest, accuracy by confidence level, the Brier score, accuracy with the least confident answers skipped, and cost and latency are reported by scripts/score.py, with the per-run detail in RESULTS.md; the prompt wording sent for each question type is not shown. data/requests/ is generated locally and gitignored, data/results/ is kept exactly as recorded so the numbers can be recomputed, and question content is used for non-commercial research only with CC BY-NC 4.0 typesetting. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
Neither is an official result and they are not comparable. The 47.8% is on the 67 items that have a key, written by the module's authors, who themselves write that the answers are not certain. The 85.4% is on the 89 items with no key at all, labelled by Claude Opus 5 alone, which the README says may be wrong. The README reports both plus a 69.2% combined figure over 156 items rather than picking one headline.
Reconstructed, and the README leads with that. SNBT papers are never released, so the 159 questions were rebuilt by the community from what participants remembered, the wording may differ from the real test and a few are flawed. The source is a 764-page recollection module rather than anything from the exam organiser, so every number measures Jev against that reconstruction.
The benchmark cannot separate the two, which is the honest problem with it. Most scored items have no module key and are labelled by a single Claude model, and the authors of the keys that do exist flag them as uncertain. What you can do is re-derive the numbers yourself: data/results/ holds one line per question and score.py reports a Brier score plus accuracy when the least confident answers are skipped.
Not to reproduce the published run. data/results/ already holds the recorded answers and the runner skips questions that already have one, so the documented commands work without a key. For a new run you write elsewhere with --out and pass the same path to score.py --results. Responses 408, 429 and 5xx are retried automatically and failed requests are logged without stopping the run; per-item cost and latency are in RESULTS.md, not the README.
Three fill-in questions have no options, so they are left out rather than inventing choices that would test the options instead of the model, which is why 67 module-keyed items are scored instead of 70. Two yes/no tables become three Noul judgments each, one per statement, reported separately because guessing there is 50% rather than 20%.
Ten figures, including line charts, a pie chart, a geometry diagram, and an operator definition. Jev reads text only, so each was transcribed into text by Claude and checked by hand; one was corrected that way when two lines first described as parallel turned out not to be once their slopes were measured.
Accuracy by subtest and by confidence level, a Brier score, accuracy when the least confident answers are skipped, and cost and latency. It can also score agreement for the Claude-labeled split, and produce a combined figure across all 156 items.
More Jev resources like this
16GitHub stars
Vendor-neutral harness that tests whether Jev reranking improves a small RAG stack, using frozen hybrid BM25 plus BGE-M3 candidates and isolating retrieval, rerank, and generation stages.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
erendikmenn/jev-rag-benchmarkPython
erendikmenn/jev-rag-benchmarkPython17GitHub stars
Local offline System One server speaking the hosted /v1/systemone wire format, so existing SDK code runs unchanged against small on-machine models.
Jev Practices & Patterns#community#awesomejev#catalog-import
Community
amithgc/local-jevPython
amithgc/local-jevPython4GitHub stars
Framework-agnostic guard layer that puts Jev in front of an agent's tool calls: a Choice risk tier, a Noul destructiveness check and a Score blast radius decide allow, deny or ask.
Jev Practices & Patterns#community#awesomejev#catalog-import
Community
lgy1027/jevshieldPython
lgy1027/jevshieldPython3GitHub stars
Python MCP server exposing Jev's three official question types plus compare, verify and batch classify, with an installer subcommand that writes the client config and a measured cost per call.
Jev Practices & Patterns#community#awesomejev#catalog-import
Community
wangkuangkuang/jev-mcp-serverPython
wangkuangkuang/jev-mcp-serverPython5GitHub stars
An inference-only prototype of a Jev-style decision model on a sub-billion-parameter hybrid backbone, returning typed probability distributions in one forward pass, with its architecture treated as an explicit guess.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
shamazharikh/qwen-rlcdPython
shamazharikh/qwen-rlcdPython1GitHub stars
A Vercel-hosted comparison of TypeSafe (Jev) against DeepSeek-flash on speed, tokens, cost and accuracy, backed by per-task Python pipelines, fixture generators and a report script. Eighteen commits, nine issues still open.
Jev Practices & Patterns#community#awesomejev#catalog-import
From the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.