User repository on GitHub
LLM Chess - evaluating Large Language Models' reasoning and instruction-following abilities by simulating chess games
- Forks
- 15
- Last pushed
- Updated Oct 2026
Repo stats from the GitHub API, cached Oct 2026.Homepage
Jev directory
CommunityJev Benchmarks & Evaluations
LLM Chess
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
- Stars
- 137
- Language
- Python
- License
- APACHE-2.0
- Reading time
- 3 min
- Published by
- Community
- Author
- maxim-saplin
- Last verified
- Oct 2026
Works forJev for Evaluation
Highlights
- Games are UCI moves in multi-turn dialogs; metrics separate chess skill (Win/Loss, Elo) from durability (game duration).
- Limits: max 200 moves, 10 dialog turns per move, and a loss after 3 illegal moves or protocol failures; logs land in _logs/ as JSON.
- Komodo Dragon levels 1-7 anchor Elo at 125 x (level + 1), from 250 to 1000; Stockfish is optional and installed separately.
- The madewithjev listing reports jev-latest played 80 games and made no illegal move; the README itself does not name Jev.
- Run with uv: uv run llm-chess for one game, run_multiple_games.py for batches, then get-refined-csv to aggregate Elo and cost.
Quickstart
bash
git clone https://github.com/maxim-saplin/llm_chess.git
cd llm_chess
uv sync
uv run llm-chessWatch out
Copy reviewed against the repo README and GitHub metadata on 2026-10-15. The Jev result comes from the madewithjev listing rather than the README, and the README publishes no per-model scores; hosted models need your own provider API keys, though local models can run through Ollama or LM Studio.
Questions about this entry
Short answers drawn from the linked project or page.
It is a benchmark that has large language models play full chess games in multi-turn dialogs against a random opponent, other LLMs, or engines like Komodo Dragon and Stockfish. It measures both strategic play and whether a model can follow the move protocol without errors.
The madewithjev build listing reports that jev-latest played 80 games and made no illegal move, which speaks to the protocol-adherence side of the benchmark. The repository README itself does not document a Jev configuration; models are wired through provider settings in .env.
Clone the repo, run uv sync to install dependencies, copy .env.sample to .env and add your provider keys, then run uv run llm-chess. By default a random player takes white against the LLM as black, and the JSON log lands in _logs/.
Wins come from checkmate or opponent errors and timeouts; draws include the 200-move limit, stalemate, insufficient material, threefold repetition, and the 75-move rule. Illegal moves and protocol mistakes are errors, and more than three in a dialog means the LLM loses.
A random player, another LLM, the Komodo Dragon engine at levels 1-7 (roughly 250 to 1000 Elo), or Stockfish. Dragon binaries go in dragon/ with llm_chess.dragon_path set, and Stockfish installs separately, for example with brew install stockfish on macOS.
The code is Apache-2.0 and free, but hosted model calls need your own API keys and are billed by your provider; each game makes many requests. The README adds a one-second throttle delay by default to avoid rate limits, and the refined CSV includes token and cost usage when the provider reports it.
Maxim Saplin maintains the repo, created in September 2024 and pushed on 2026-10-14. Results appear on a live leaderboard at maxim-saplin.github.io/llm_chess, and the method is written up in a NeurIPS FoRLM 2025 paper, arXiv 2512.01992.
More Jev resources like this
A minimal coding agent that filters every tool result through Jev before the main model reads it, plus the A/B harness that measured pass@1 and cost. The final TB4 follow-up closed the experiment without an end-to-end win.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
astro-han/jev-harnessPython
astro-han/jev-harnessPythonA benchmarking studio and real-time simulator that pits Jev (remote TypeSafe System One), a local Laya model and the optimal blackjack Basic Strategy against each other on identical deterministic shoes. It scores every deviation turn by turn and runs headless batches of up to 100,000 rounds.
Jev Benchmarks & Evaluations#community#github#blackjack
Community
mrbanana55/decision_blackjackPython
mrbanana55/decision_blackjackPythonA benchmark for how efficiently decision models plan across multi-step problems: models that explore fewer unnecessary intermediate states score higher, and a perfect score means beelining along a shortest path. Seven domains, seeded trials and classic baselines.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
kyle-pena-nlp/beelinebenchPython
kyle-pena-nlp/beelinebenchPythonBenchmark harness that scores TypeSafe Jev against any OpenRouter model or local checkpoint on labelled classification data: accuracy, calibration (ECE), confidence distribution, latency and cost across five tasks with 300 items each.
Jev Benchmarks & Evaluations#community#catalog-import#cobanov
Community
4esv/jev-evalPython
4esv/jev-evalPythonA reproducible 100-task benchmark comparing Jev with Qwen3.5 0.8B, 2B and 4B across three thinking budgets: Jev scored 77.0%, the 2B model at 2,048 tokens 82.0%, and the 4B 96.7%.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
softpudding/jev-frontier-100Python
softpudding/jev-frontier-100PythonA Python research harness evaluating Jev (TypeSafe System One, jev-1.13.0) as a relevance judge for task-level tool spaces. One API call scores every tool with a yes/no noul question per tool, and the repo benchmarks keep-all versus per-request, per-task and trace-based spaces on TGB and BFCL.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
xuan7zhang/jev-toolspacePython
xuan7zhang/jev-toolspacePythonFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.