Jev directory
CommunityJev Benchmarks & Evaluations
System-One Control Bench — socb.dev
A benchmark asking how well individual decisions add up to a completed task: decision and chat models steer an agent through 100 grid puzzles, where Jev chooses an optimal move in about 90% of fixed exam positions yet finishes only 57 of 100 games.
- Reading time
- 3 min
- Published by
- Community
- Last verified
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is System-One Control Bench — Exemocaro.
Highlights
- 100 grid puzzles in two conditions: Map only, and Full context adding surroundings, move history, move outcomes and a subgoal.
- Jev (jev-1.13.0 via its API): 57 of 100 games won with full context and 32 with map only, mean progress 0.68 and 0.43.
- Leaderboard leader DeepSeek V4.1 Flash (reasoning) wins 80 of 100 with full context; Gemma 4 26B 61; Qwen3.5-4B 45; GLiClass 13; Laya 12.
- Baselines: an optimal solver 100/100, greedy strategies 28-37/100 and random moves 4/100; the move limit is twice the shortest route.
- The wider study tests ten context conditions and move rules of 2 or 3 steps at once, from 16 to 84 options per move.
Watch out
Page text fetched 2026-10-17. Scores are the site's own results on a small fixed benchmark, reported with 95% intervals for some columns; Jev's reported API cost is not listed and cost figures cover 200 games across both conditions, excluding local computing. The capture is the landing page - leaderboard details like SPL and time per answer, plus the report and code, were not fetched in this digest.
Questions about this entry
Short answers drawn from the linked project or page.
Whether individual decisions add up to a completed task. It tests whether a decision model can guide an agent through a small grid puzzle and compares it with chat models and simple programmed strategies on the same 100 puzzles.
Reach the goal, avoid walls, and collect a key when a locked door blocks the way. At each move the model sees the puzzle as text, chooses from a list of moves, then sees the new situation and chooses again, so finishing takes many useful decisions in a row.
With full context, Jev chooses an optimal move in about 90% of the positions in a separate fixed exam, but it finishes only 57 of 100 whole games (32 of 100 with map only), showing that strong single decisions do not guarantee a finished game.
Map only gives the rules, complete map, agent position and whether it carries the key. Full context adds code-generated explanations: surroundings, move history, what each offered move would do, and the next target. The study also runs ten conditions varying which components are present.
DeepSeek V4.1 Flash with reasoning leads at 80 of 100 games won; Gemma 4 26B 61, DeepSeek V4.1 Flash without reasoning 60, Jev 57, Qwen3.5-4B 45, GLiClass 13 and Laya 12. Programmed baselines bracket them: optimal solver 100, greedy 28-37, random 4.
It is small and fixed, at 100 puzzles with a move limit of twice the shortest route, and the site reports 95% intervals because scores depend on which puzzles are in the set. Jev's cost is not reported, and local computing is excluded from cost figures.
The puzzles, recorded games, code and analysis are public, and the page offers two saved games to replay one move at a time with no new model requests. It links a report, code and data, and a form to submit a model; Jev ran as jev-1.13.0 through its API.
More Jev resources like this
A deterministic gridworld that benchmarks System One decision backends, typed calibrated decisions over a declared option set with one forward pass and no decoding, in a sequential loop, with a scripted oracle scoring agreement without labels.
Jev Benchmarks & Evaluations#community#github#benchmark
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonA benchmark of Jev and other decision models on trolley-problem states, with runners, raw answers and analysis in one repository. The v1 snapshot covers 1,098 states and the October 2026 v2 run has 14 models on 5,240 frozen states.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
zachlandes/trolley-benchHTML
zachlandes/trolley-benchHTMLIndependent measurements of typed-decision models: Jev (TypeSafe API) against Laya (open weights) on 40 Chinese support-ticket classifications, plus a local-first cascade that matches Jev's accuracy at about 1.8x the speed.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
yibie/laya-jev-labPython
yibie/laya-jev-labPythonThe code and data behind ProbabilityBench: text-based simulated worlds with exact class probabilities, so a decision model's reported probabilities can be compared directly with ground truth. An arXiv paper and a Hugging Face dataset accompany it.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
ishida-lab/probabilitybenchPython
ishida-lab/probabilitybenchPythonA reproducible experimentation framework and report series for Jev: it runs shared tasks at scale across benchmarks, routing and numeric prediction, and publishes bilingual findings on where the model holds up and where it does not.
Jev Benchmarks & Evaluations#community#python#evals
Community
keta1930/what-the-jevPython
keta1930/what-the-jevPythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.