Docs
CommunityJev Benchmarks & Evaluations
next-jev Tetris Demo
One timed Tetris game by the next-jev choice-head model on Qwen3.5-0.8B: 1,000 pieces, 398 lines and 9.52 seconds on an H100 NVL, with per-piece timings and decoded latent reasoning.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- JonesLin
- Last verified
- Oct 2026
Works forJev for Realtime control
Used inJev for Gaming
- Capabilities
- Open source
Highlights
- Checkpoint stage2-choice/checkpoint-00006056 is based on Qwen3.5-0.8B; seed 42 ran up to 1,000 pieces in one loop.
- The game cleared 398 lines and stopped at the piece limit without topping out, in 9.52 seconds on an NVIDIA H100 NVL.
- Model time per piece averaged 3.75 ms and whole-step time 9.19 ms; bf16, fused projections, torch.compile and CUDA graphs were used.
- The card warns the decoded reasoning text is not reliable and is included for analysis only.
- Reasoning is decoded from 128 hidden workspace vectors by a weak aux_decoder that predicts about 40% of next words with previous words given.
Watch out
The dataset card states no licence, and its own warning dominates: the decoded reasoning text does not match the board, names the chosen move no more often than a random other move, and the stage 2 decoder was judged reasonable on 0 of 366 steps and partly reasonable on 20. The game itself is a single seed-42 sample.
Questions about this entry
Short answers drawn from the linked project or page.
The next-jev choice-head model at checkpoint stage2-choice/checkpoint-00006056, based on Qwen3.5-0.8B, playing seed 42 for up to 1,000 pieces in one loop.
It reached the 1,000-piece limit without topping out, cleared 398 lines, and finished in 9.52 seconds on an NVIDIA H100 NVL with bf16, fused projections, torch.compile and CUDA graphs.
Model time from token ids to option scores averaged 3.75 ms with p95 3.80 ms, and whole-step time including localhost HTTP overhead averaged 9.19 ms with p95 10.90 ms.
No. The card warns the decoded reasoning is not reliable: it does not match the board or explain the choices, and it names the chosen move no more often than a random other option.
The model decides in 128 hidden workspace vectors and never writes text; after the game, each saved request is re-run and a small aux_decoder greedily decodes text from the last loop's workspace.
The same game with the newer last-block decoder over 366 pieces, a raw slow video with decoded reasoning for every piece, and an LLM judge verdict per step: 0 of 366 reasonable and 20 partly reasonable.
game/ holds one JSONL row per piece with piece, candidates, chosen id, heuristic rank, confidence, lines and timings, plus CSV and summary JSON, and requests captures the exact model input for each piece.
More Jev resources like this
A four-step experiment asking whether Jev, a fast model that picks among named options, can act as the policy in StarCraft multi-agent micro (SMAC), comparing every step against baselines that had the same information but no Jev.
Jev Benchmarks & Evaluations#community#github#benchmark
Community
wang90063/jev-on-smacPython
wang90063/jev-on-smacPythonA benchmark that steers decision models through 100 key-and-door grid puzzles with one Jev Choice per move, scored against an exact BFS solver; it found Jev picks an optimal move about 90% of the time yet finishes only 57 of 100 games.
Jev Benchmarks & Evaluations#community#benchmark#gridworld
A Python benchmark that evaluates LLMs' reasoning and instruction-following by making them play chess against a random player or the Komodo Dragon engine in multi-turn dialogs, separating chess skill from protocol durability.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
maxim-saplin/llm_chessPython
maxim-saplin/llm_chessPythonProphetLab's Texas Hold'em benchmark and playground for decision models: separate cash and sit-and-go leaderboards, hand replays, a Jev-backed advisor, and bring-your-own-agent tables, all with virtual chips.
Jev Benchmarks & Evaluations#community#catalog-import#madewithjev
Community
prophetlab/jevpokerbenchPython
prophetlab/jevpokerbenchPythonA harness for evaluating TypeSafe Jev as a policy in Gymnasium and Atari environments: adapters turn observations into structured JSON and Jev picks a legal action through a single Choice question, with no training.
Jev Benchmarks & Evaluations#community#catalog-import#logicrw
Community
taodav/jev_deep_rlPython
taodav/jev_deep_rlPythonNotebook benchmark that compares Jev with open models and specialised tools on five document-pipeline chores: language detection (Choice(12)), page orientation (Choice(4)), RVL-CDIP classification (Choice(16)), bundle splitting and parse-tier triage, with liteparse doing all PDF rendering and OCR work.
Jev Benchmarks & Evaluations#community#evaluation#document-processing
Community
run-llama/jev_vs_ossJupyter Notebook
run-llama/jev_vs_ossJupyter NotebookFrom the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.