Can LLMs Do What Jev Does? A Replication Attempt

That is essentially the point. He rebuilt the JSON demo with Gemma3 270M: fixed answer candidates mapped to single tokens, read only the logits at the answer position, batch independent questions, reuse the KV cache for a shared prefix, and assemble the JSON in code. The 77x is roughly proportional to the output tokens saved, and Jev is not involved.

Both, and he flags the gap himself. On his machine Jev averaged 276 ms and reached distance 1,226, against Llama 3.1 8B at 356 ms and 1,138. But he measured roughly 3x the latency of the public demo and thinks that difference alone could explain the worse play, so latency and outcome are confounded in this run.

No, and the article says so. The logit trick needs an API that returns logits or logprobs, and most strong models now are reasoning models that do not expose them. So the comparison is mostly older non-reasoning models, including Llama 3.1 8B and Gemini 2.5 Flash-Lite, and he explicitly declines to claim Jev beats today's best.

Less seriously than the vendor pitch. He does reproduce the vendor's argument that plain LLM logits carry position bias, tokenizer artifacts, and overconfidence. But he then links a commenter's test where Jev's Choice answered "1" on all 400 hidden-die rolls at an average 83% where the true rate is one in six, and his conclusion is to be careful about reading the output as a real-world probability.

Map each fixed answer candidate to a single token, then read only the logits at the answer position in one forward pass instead of generating JSON token by token. Independent questions can be batched, a shared prompt prefix can reuse the KV cache, and the JSON is assembled in code. The reported 77x is roughly proportional to the output tokens saved, and Jev is not involved at all.

Closer than the headline suggests. Jev reached distance 1,226 at 276 ms; Llama 3.1 8B 1,138 at 356 ms, Gemma 4 26B A4B 1,127 at 572 ms, Qwen3 235B A22B 686 at 593 ms, and Gemini 2.5 Flash-Lite 296 at 719 ms. The author notes Llama and Gemma land quite near Jev's score, and that only non-reasoning models exposing logprobs could be tested.

The article thinks it may shrink. It cites a measurement of DeepSeek V4 Flash at roughly 90 ms time to first token on a single NVIDIA B300, and notes that fast non-reasoning models are already entering the ~100 ms range. Its conclusion is that if OpenAI or Anthropic exposed a similar typed interface over a plain LLM API, much of this could be replicated, so the specialization advantage is not yet proven durable.