Jev as an LLM Judge

Not well. One reply says it may simply be selecting the largest number, another argues the choice came from the model name's self-evident wording, and a third calls gpt-6-Astra a bad answer that fails simple web UI work. Read the pick as an API demo, not a ranking.

Nothing measurable follows. There is no fixture set, no pairwise comparison, and no agreement or correlation figure against human labels in the post. What is shown is one preference among candidates, which demonstrates the request shape rather than judge quality.

From the author saying they tried a bunch of prompts and still did not burn through $0.10. That is a spend anecdote on one prompt sweep, not a per-judgement cost, and the 20-200x and 40-400x figures in the post are TypeSafe's launch numbers rather than anything measured here.

The rubric, not the budget. Jev returns typed decisions with probabilities over options you define, so criteria and an answer key have to come from you. The post does not describe calibrating a threshold, nor how to handle disagreement between two judges on the same item.

Yuchen Jin, who works on AI systems and products at Databricks and was previously co-founder and CTO of Hyperbolic and OctoAI, which Nvidia acquired, with Apache TVM and a PhD behind him. No stake in Jev is disclosed, and the post is a quick API demo rather than a vendor promotion.

Which model is AGI, chosen from a set of candidates. It is a single preference question of the kind LLM-as-a-judge is used for, and the screenshot shows Jev picking one. The post does not say how many candidates were offered or how the option list was framed, and both affect a Choice.

It only notes the potential, saying this could make LLM-as-a-judge insanely fast and nearly free. He does not report running a judging workload, publishing agreement numbers, or recommending it for a specific use, so the post is a prompt-based reaction to the API rather than a verdict on judge quality.