Question 1

What is the benchmark's main finding?

Accepted Answer

Strong single decisions do not add up to a finished task: Jev picks an optimal move in about 90% of fixed positions yet finishes only 57 of 100 games against the exact solver's scoring.

Question 2

Which other players can I compare against Jev?

Accepted Answer

The repo includes random, greedy, greedy-walls and solver baselines, local Laya, GLiClass and Qwen3.5-4B players, and OpenRouter chat models such as Gemma 4 26B and DeepSeek V4.1 Flash.

Question 3

How do I run the benchmark locally, including paid players?

Accepted Answer

uv sync, then uv run socb benchmark runs the free players on all 100 puzzles in a few seconds, uv run socb exam runs the fixed-state exam where the four free baselines answer 495 positions, and uv run socb web serves a board viewer. Paid players need TYPESAFE_API_KEY and/or OPENROUTER_API_KEY in .env and are refused without --allow-paid, which prints the maximum call count the run can make. Local models need uv sync --extra local, which installs several GB of PyTorch.

Question 4

How is a move scored, and what do the four scores mean?

Accepted Answer

A breadth-first solver knows the shortest route from every position, so each move either starts a shortest route or it does not, which makes the scoring exact rather than model-judged. won is games finished within twice the shortest route; progress is how much closer the agent got at its best point, 1 meaning finished; SPL is shortest-route moves over moves used for a won game; and errors counts games the player failed to answer, which count as lost and should be zero.

Question 5

What makes the puzzles hard, and what can I vary between runs?

Accepted Answer

The 100 puzzles are YAML files spread over solver-distance levels 1, 2, 3, 4, 5, 6, 8, 10, 12, 15 and 20, with # wall, . floor, A agent, G goal, K key and D locked door. Ten information conditions all include the map and add surroundings, memory, lookahead and subgoal in different combinations, with everything as the full context. Five rule sets run from compass at 4 options to up-to-three-moves at 84, where a blocked step is wasted but the remaining steps still run.

Question 6

How do I add my own model to the benchmark?

Accepted Answer

Any chat model on an OpenAI-compatible endpoint, or a decision endpoint, is added with one entry in players.toml and no code. A chat model on OpenRouter needs only a line in LLM_MODELS in players/remote.py. A Python player subclasses Player, implements choose to return a Choice, and registers itself in PLAYERS, with paid=True if it costs money; in the repository that registration goes in src/system_one_control/players/__init__.py.

Question 7

Is there a cheaper way to compare models than a full run?

Accepted Answer

Yes. The core leaderboard track is compass rules over the map and everything conditions on all 100 puzzles, which is 200 games per player instead of the full 5,000, so models can be compared without paying for everything. Every move of every game is written to benchmarks/__.jsonl with a .txt score table beside it, the analysis scripts recompute the paper's tables and figures with no API calls, and single runs can be scoped further with --levels, --rules and --conditions.

System-One Control Bench

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community