User repository on GitHub
How efficiently do decision models navigate multi-step problems?
- Forks
- 1
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.
Jev directory
CommunityJev Benchmarks & Evaluations
BeelineBench
A benchmark for how efficiently decision models plan across multi-step problems: models that explore fewer unnecessary intermediate states score higher, and a perfect score means beelining along a shortest path. Seven domains, seeded trials and classic baselines.
- Stars
- 3
- Language
- Python
- License
- APACHE-2.0
- Reading time
- 2 min
- Published by
- Community
- Author
- Kyle Pena
- Last verified
- Oct 2026
Works forJev for Evaluation
Used inJev for Software & DevTools
Highlights
- Scores planning efficiency by how few unnecessary intermediate states a decision model explores; a perfect score means it beelined along a shortest path.
- Seven domains: 8-puzzle, Blocksworld, Countdown, word ladder, Wikispeedia, Rush Hour and keys and doors, each with heuristic, random and oracle baselines.
- In benchmark 1.0.0 Jev 1.13 scored 0.26 overall and beat the heuristic on 3 of 7 domains; Mercury Decide and pplx-decider 1.0 led at 0.28.
- Trials are randomly generated, and `--seed N` reproduces the same set across machines; a container image accepts any Jev-protocol or OpenAI Decisions API model.
- The overall score is the geometric mean of the domain scores; the README notes runs that did not solve within 2,500 nodes mean true scores are lower.
Watch out
Grounded on the truncated README and GitHub metadata reviewed 2026-10-15; the quick-start section was cut off, so install and run commands are not quoted here. Results are benchmark 1.0.0 with at least one unsolved run per model marked with asterisks, so several scores are upper bounds.
Questions about this entry
Short answers drawn from the linked project or page.
How well a decision model plans while solving a multi-step problem like Blocksworld or a sliding tile puzzle, scored by how few unnecessary intermediate states it explores to reach the goal.
In benchmark 1.0.0 Jev 1.13 scored 0.26 overall and beat the classic heuristic in 3 of 7 domains. Mercury Decide and pplx-decider 1.0 led the table at 0.28.
Seven: 8-puzzle, Blocksworld, Countdown, word ladder, Wikispeedia, Rush Hour and keys and doors. Each has the classic heuristic of the domain, random choice and an oracle baseline.
It ships on PyPI with Python 3.11+ and as a container image, per the badges. The quick-start section was outside the reviewed excerpt; docs/usage.md covers the optional --seed N master seed.
Any model that uses the Jev protocol or OpenAI's Decisions API, per the README, including hosted decision models and local ones like Liquid d1 and GPT-6 Luna (Decisions) in the 1.0.0 table.
At least one run of that model or heuristic did not solve within 2,500 nodes, so the reported score is an upper bound and the true value is lower. The overall score is the geometric mean of the domain scores.
No. A note says classic state-space search such as A* will almost certainly remain most efficient for classic problems, and BeelineBench is meant to evaluate planning ability, not to replace those techniques.
More Jev resources like this
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA minimal coding agent that filters every tool result through Jev before the main model reads it, plus the A/B harness that measured pass@1 and cost. The final TB4 follow-up closed the experiment without an end-to-end win.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
astro-han/jev-harnessPython
astro-han/jev-harnessPythonHead-to-head benchmark for System One-style classification models across choice, noul and score questions. Jev led both hash-locked suites at 0.974 micro accuracy on v1 (8 tasks, 78 cases) and 0.964 on v2 (49 tasks, 866 cases), ahead of local Von 1.1, GLiNER2 and Laya.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
jabr/classifier-benchmarkPython
jabr/classifier-benchmarkPythonBenchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonA Python research harness evaluating Jev (TypeSafe System One, jev-1.13.0) as a relevance judge for task-level tool spaces. One API call scores every tool with a yes/no noul question per tool, and the repo benchmarks keep-all versus per-request, per-task and trace-based spaces on TGB and BFCL.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
xuan7zhang/jev-toolspacePython
xuan7zhang/jev-toolspacePythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.