User repository on GitHub
100 original tasks comparing Jev with Qwen3.5 0.8B, 2B and 4B across three reasoning budgets; reproducible results and token logprobs.
- Forks
- 0
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
Jev Frontier 100
A reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
- Stars
- 1
- Language
- Python
- License
- MIT
- Reading time
- 2 min
- Published by
- Community
- Author
- softpudding
- Last verified
- Oct 2026
Highlights
- Jev 1.13.0 scored 77.0%; Qwen3.5 2B with a 2,048-token thinking budget scored 82.0% and 4B with the same budget 96.7%.
- Budget matters: 4B with thinking off scores 56.0% and at 512 tokens 78.3%, so comparisons need the thinking condition attached.
- 100 questions across ten domains, including customer service, policy rules, formal logic, mathematics, temporal reasoning and code semantics.
- All 3,000 retained responses produced valid final answers; no majority vote or best-of selection was used.
- Published conditions were chosen after seeing results and the release is called exploratory; all questions, gold answers and trials ship for offline recompute.
Quickstart
bash
python -m pip install -e '.[plots]'
python -m unittest discover -s tests -v
python -m jf100.budget_report v0.2-budget --from-publicWatch out
Grounded in the truncated README, GitHub metadata and result tables in the digest reviewed on 2026-10-15; the README says the benchmark measures a synthetic English four-choice task set under specified budgets, not a universal intelligence ceiling, and that difficulty labels are uncalibrated and the dataset has had no independent human expert review.
Questions about this entry
Short answers drawn from the linked project or page.
A small reproducible benchmark of 100 original four-choice questions comparing Jev 1.13.0 with Qwen3.5 0.8B, 2B and 4B across three thinking budgets. Jev scored 77.0%, and the README publishes results, token logprobs and reproduction steps.
Jev 1.13.0 scored 77.0%. Qwen3.5 2B with a 2,048-token thinking budget scored 82.0%, and 4B with the same budget 96.7%. Each condition is 100 questions times 3 trials, and no majority vote or best-of selection was used.
No. The README says the result needs its thinking condition attached: 4B with thinking off scores 56.0% and at 512 tokens 78.3%. It also says the benchmark does not establish a universal intelligence ceiling or parameter equivalence.
Ten domains with ten questions each: customer service, policy rules, discourse, formal logic, relations, mathematics, temporal reasoning, code semantics, algorithms and evidence integration. There are 50 counterfactual pairs and 60 executable-oracle answers.
Python 3.10+ and no third-party grading dependencies: install the package, run the unit tests, then recompute all published scores, paired intervals and confidence diagnostics offline from the shipped data with the budget_report command.
No. The README states the published conditions were selected after observing experimental results and calls the release exploratory rather than blind preregistration; original protocol and config hashes are in data/release_provenance.json.
Yes: pull the Q8_0 Ollama tags, point JF100_LLAMA_SERVER at a llama-server build supporting reasoning_budget_tokens and token logprobs, set JEV_KEY, and run the jf100 module. Seeds 101/202/303 rotate option positions per trial.
More Jev resources like this
Benchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA measured audit of 300 public pages for AI citability using nine Jev decisions per page, checked against an LLM judge: Jev agreed on 85.8% of 900 decisions and cost 158 times less per page, with latency measured for both.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
stas4000/jev-geo-auditPython
stas4000/jev-geo-auditPythonA Python research harness evaluating Jev (TypeSafe System One, jev-1.13.0) as a relevance judge for task-level tool spaces. One API call scores every tool with a yes/no noul question per tool, and the repo benchmarks keep-all versus per-request, per-task and trace-based spaces on TGB and BFCL.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
xuan7zhang/jev-toolspacePython
xuan7zhang/jev-toolspacePythonA Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonIndependent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonA reproducible lab measuring how several coding agents coordinate on one repository, plus Médula, a kernel that asks TypeSafe Jev before every write whether a change collides with what the other agents are doing.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
joaquinruiz/medulaPython
joaquinruiz/medulaPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.