User repository on GitHub
- Forks
- 2
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Classifier Benchmark
Head-to-head benchmark for System One-style classification models across choice, noul and score questions. Jev led both hash-locked suites at 0.974 micro accuracy on v1 (8 tasks, 78 cases) and 0.964 on v2 (49 tasks, 866 cases), ahead of local Von 1.1, GLiNER2 and Laya.
- Stars
- 3
- Language
- Python
- Reading time
- 2 min
- Published by
- Community
- Author
- jabr
- Last verified
- Oct 2026
Highlights
- Two hash-locked suites: v1 covers 8 tasks and 78 cases, v2 covers 49 tasks and 866 cases, and `just validate` checks each against cases/hashes.json.
- Jev scored 0.974 micro accuracy on v1 (~302 ms, ~$0.000014/call) and 0.964 on v2 (~330 ms), ahead of Von 1.1, GLiNER2 and Laya.
- Test cases are plain TOML with the schema in bench/cases.py, so harnesses in other languages can read them; suites are locked once with `just lock <suite>`.
- Models under test: Von 1.1, GLiNER2, GLiNER2.5-Decide, Laya and jeff (GLiFormer large 576M) locally, plus Jev hosted via OpenRouter.
- All cases are synthetic, generated and cross-checked by a committee of LLMs; the README notes models may incorporate the public cases into training data.
Quickstart
bash
uv sync
uv run python -m bench.run --backend jev --suite v1 --out results/jev-run.jsonWatch out
Grounded on the truncated README and GitHub metadata reviewed 2026-10-15; the usage section is cut off mid-word and the license is listed as NOASSERTION. Results are the project's own runs on Apple MPS with synthetic cases, so treat the figures as one harness's report rather than independent validation.
Questions about this entry
Short answers drawn from the linked project or page.
A head-to-head benchmark for System One-style classification models. Every model answers the same question JSON - choice, noul or score - on the same tasks, and results cover accuracy, latency and cost.
Jev led both suites: 0.974 micro and 0.972 macro accuracy on v1 (8 tasks, 78 cases) at ~302 ms and ~$0.000014 per call, and 0.964 micro / 0.966 macro on v2 (49 tasks, 866 cases) at ~330 ms.
Jev (typesafe/jev-1.13 via OpenRouter) against local models: Von 1.1, GLiNER2, the decision-tuned GLiNER2.5-Decide, Laya and jeff (GLiFormer large 576M, installed with an optional extra).
Clone the repo and use uv: `uv sync` installs the core backends, weights come from `just download <org>/<name>`, and a suite runs with `uv run python -m bench.run --backend jev --suite v1 --out results/jev-run.json`.
The local backends (Von, GLiNER2, Laya, jeff) run free on your machine; the reported runs used Apple MPS. The Jev backend is hosted and needs a key, and the README adds `uv sync --extra jeff` for the jeff backend.
Yes. The README states the benchmarks are public and models are free to incorporate the test cases into training data, which is worth remembering when comparing published numbers.
The cases are synthetic, generated and cross-checked by LLM committees, and the numbers are the project's own runs on Apple MPS. The README's usage section is truncated and the GitHub license is listed as NOASSERTION.
More Jev resources like this
Benchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA measured audit of 300 public pages for AI citability using nine Jev decisions per page, checked against an LLM judge: Jev agreed on 85.8% of 900 decisions and cost 158 times less per page, with latency measured for both.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-geo-auditPython
stas4000/jev-geo-auditPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonA Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonIndependent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonA reproducible lab measuring how several coding agents coordinate on one repository, plus Médula, a kernel that asks TypeSafe Jev before every write whether a change collides with what the other agents are doing.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
joaquinruiz/medulaPython
joaquinruiz/medulaPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.