Giamme/apple-silicon-systemone-benchmark
User repository on GitHub
Open-source Apple Silicon benchmark for System One decision models
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Benchmarks & Evaluations
Apple Silicon Systemone Benchmark
Native Apple Silicon harness comparing local decision models through one decision interface, running models serially and emitting an offline dashboard with latency and accuracy.
- Stars
- 1
- Language
- Python
- License
- APACHE-2.0
- Published by
- Community
- Author
- Giamme
- Last verified
- Oct 2026
About this project
An open-source benchmark for local decision models and compatible services on native Apple Silicon macOS. It compares the same labeled cases through a common decision interface, runs managed models one at a time, and produces an offline HTML dashboard with CSV, JSON, Markdown, and request-level records.
Quoted from the project's README. Read the full README.
Highlights
- Requires native arm64 macOS, uv plus git, Python 3.11+, and gigabytes for weights.
- Harness is stdlib-only; weights come from upstreams under their own licenses.
- Run via benchmark.sh all, setup, run, validate, and models list with profile or model filters.
- Outputs summary HTML, CSV, JSON, Markdown, logs, and request-level records per run.
Watch out
Included 24-case set checks workflow only and is not evidence for production; keep private datasets outside the repo. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
No. The 24-case synthetic set only validates the pipeline. Bring representative labeled traffic with ambiguity, negation, long inputs, and your real labels, and disclose dataset, hardware, revisions, and failures when sharing.
Full runs Laya, Von, Kev 0.8B, and Rizzo Flow 4B Q4_K_M; light runs Laya and Kev. Profiles never infer quality from chip generation, and on low unified memory start with light if full cannot load every model.
Use JSONL with id, state, questions, and expected per line, then validate before setup. Reports record filename, case count, and SHA-256 only, results and models/local stay Git-ignored, and non-loopback endpoints need --allow-remote.
Warm p50, p95, and p99 are end-to-end HTTP latency after warmup with cache-busting IDs, plus accuracy, Brier, ECE, and error-budget coverage. RSS is a host-memory proxy, not Metal allocation, and failed models still appear with nonzero exit.
macOS running natively on Apple Silicon (arm64), not Rosetta, with uv and git for built-in model setup, Python 3.11 or newer, and several gigabytes of disk space for downloaded environments and weights. The harness itself uses only the Python standard library.
One JSON object per line with id, state, questions and expected. state can be a nonempty string or object, every question needs a labeled expected answer, and the supported types are choice, noul (boolean) and score (a zero-based index into criteria).
A results/<timestamp>/ directory with a standalone summary.html dashboard, summary.csv, summary.json, summary.md and non-identifying system.json, plus per-model server logs and requests and decisions JSONL files. Reports show the dataset filename, case count and SHA-256, never its contents or absolute path.
More Jev resources like this
1GitHub stars
Eleven-level Jev course from hello-world to a ticket-triage capstone, pairing short lessons with runnable scripts against the live API and using LiteLLM so the writing half works with any provider.
Jev Benchmarks & Evaluations#community#github
Community
nadeemcite/jev-crash-coursePython
nadeemcite/jev-crash-coursePython1GitHub stars
Wire-compatible Jev re-implementation reading typed decisions from an LLM's next-token logits, plus a reproducible JevBench-based comparison of that approach against Jev on identical items.
Jev Benchmarks & Evaluations#community#github
Community
dashbi1/jev-simPython
dashbi1/jev-simPython0GitHub stars
Eleven-level Jev crash course from zero with runnable examples, LiteLLM write-half, and a ticket-triage capstone with tests and evals.
Jev Benchmarks & Evaluations#community#github
Community
nadyth/jev-crash-coursePython
nadyth/jev-crash-coursePython338GitHub stars
SREGym stands up live Kubernetes clusters so SRE agents can be evaluated against 90 fault scenarios, including OS-level faults and metastable and concurrent failures, in containers that cannot see the grading logic.
Jev Benchmarks & Evaluations#community#jevbest
Community
SREGym/SREGymPython
SREGym/SREGymPython217GitHub stars
Standardizes hundreds of public datasets into interchangeable multiple-choice, classification, and token-classification templates, and deterministically recasts them into typed choice, score, and noul decision requests.
Jev Benchmarks & Evaluations#community#jevbest
Community
sileod/tasksourcePython
sileod/tasksourcePython15GitHub stars
A BEIR reranking recipe that asks up to 30 parallel Noul relevance questions in one Jev state, with RESULTS.md against cross-encoders and hosted rerankers.
Jev Benchmarks & Evaluations#community#evaluation#python
Community
hev/rerankerPython
hev/rerankerPythonFrom the Jev community
Posts from builders shipping with Jev right now.