Are the chess numbers real, or is the harness playing for Jev?

It depends on the configuration, and the README labels it. Code enumerates the legal moves so an illegal move is impossible, but in the --filter-blunders games code also plays any available checkmate and drops moves that hang a piece before Jev chooses. Those runs are marked hybrid, and the single win over depth-1 Stockfish needed exactly that setup.

Is it really worse than random when you just hand it a board?

From a FEN string, yes: 533 centipawns of mean loss against a 403 random baseline, with 73% of moves blunders. An ASCII board plus move history gives 409, code-computed facts give 144, and adding one-ply tactical facts gives 90 with blunders down to 13%. How the facts are worded mattered as much as computing them.

Why not score every move and take the highest?

It was measured and it is worse. One Score per move landed at 197 centipawns of mean loss against 144 for a single Choice across all legal moves, and cost four times the tokens, though the scores did correlate 0.46 with Stockfish. Splitting the decision into piece-then-square was worse again at 217.

Is the addressee result a fair comparison?

It is a real task with a baseline attached: precision is 1.0 across the board, F1 is 0.96 on clean text and 0.93 on lowercase, punctuation-free transcripts with misheard names, and exact-set accuracy is 92% against 64% for a fuzzy name-matching heuristic. Latency is quoted at 0.17 s median with 8 to 10 questions per call.

How much did wording change the tactical results?

The first tactical version scored worse than rich at 241 centipawns because the facts were right but the wording was noise: every move carried a leaves-hanging note even when the pawn was already hanging. Reporting only deltas the move causes, leading with the material verdict in plain words, and suppressing threat notes on moves whose piece does not survive took it from 241 to 90.

How fast is one chess move?

Over 425 game moves at the tactical level, the single Choice call over a median of 32 legal moves and about 2,100 input tokens took a median 166 ms, p90 256 ms and p99 403 ms. Computing the tactical facts in python-chess adds about 9 ms, so a move is roughly 175 ms end to end and a 40-move game about 7 seconds of its own thinking time.

How was the Elo estimate built?

Stockfish's calibrated strength floor is UCI_Elo 1320 and Jev sits below it, so there was no off-the-shelf opponent with a known rating in range. Instead seven bots played 40 games per pair, 440 bot-vs-bot games with no Jev tokens, and their ratings were fitted by maximum likelihood under the Elo model with Stockfish-at-1320 pinned as the anchor.

CommunityJev Practices & Patterns

Jev Benchmark — wondertwins

Two committed benchmarks of Jev: chess, deliberately outside its lane, and NPC addressee detection, squarely inside it, with raw requests and a playground.

See highlights

Open github.com

Stars: 7
Language: Python
License: MIT
Published by: Community
Author: wondertwins
Last verified: Oct 2026

wondertwins/jev-benchmark

User repository on GitHub

View on GitHub

Benchmarks and a playground for TypeSafe's Jev (System One) model: chess, and who-is-the-player-talking-to for speech-to-text game NPCs

Forks: 1
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

This is one of several Jev listings for the same project. The canonical entry is Jev Benchmark — YidiDev.

About this project

Two hands-on benchmarks of Jev, the "System One" model from TypeSafe. Jev doesn't generate text or reason step by step. You hand it state (a JSON blob) and typed questions (yes/no, pick-one, or rate-on-a-scale) and it returns calibrated probabilities in about 200 ms. The pitch is "programmable common sense": code owns the workflow, Jev supplies the judgment. I wanted to know where that holds up and where it breaks, so I tested it on two very different tasks:

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification Jev for Evaluation

Used inJev for Software & DevTools

Languages: Python

Highlights

From a FEN string alone it is worse than random at 533 centipawns of mean loss against a 403 baseline; code-computed facts bring that to 144.
One Choice over all legal moves is both the cheapest and the best method at 144, beating one Score per move at 197 with four times the tokens.
Addressee detection reaches F1 0.96 on clean text and 0.93 on misheard, unpunctuated transcripts, with 92% exact-set accuracy against 64% for fuzzy matching.
A fitted Elo ladder puts one-ply tactical play near 950, checking out every weaker bot but losing to depth-1 Stockfish.

Watch out

The chess ladder is the author's own construction: seven bots were rated by maximum likelihood with Stockfish at 1320 pinned as the anchor, so 950 is an estimate against a synthetic scale rather than a published rating. Games marked hybrid had code play any available checkmate and remove moves that hang a piece before Jev chose, and the only win over depth-1 Stockfish came in that configuration. All figures come from one served model version, and the repository's own conclusion is that Jev is a judgment engine, not a calculator. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.