CommunityJev Benchmarks & Evaluations
TypeSafe's Jev Can't See. I Made It Guess What I Drew Anyway — mikulskibartosz.name
A preregistered Quick, Draw! study: 400 SVG-sketch classifications with Jev versus Sonnet on the same text, plus base64 and image baselines.
- Format
- Article
- Published by
- Community
- Author
- Bartosz Mikulski
- Last verified
- Oct 2026
This is one of several Jev listings for the same project. The canonical entry is TypeSafe's Jev Can't See. I Made It Guess What I Drew Anyway — news.ycombinator.com.
Highlights
- 19 September 2026. Jev on SVG scored about 35% accuracy versus Sonnet-on-SVG about 57% on the same 400 drawings.
- Jev answered airplane for 209 of 400 sketches; per-object accuracy ranged from 0/40 on cats to 34/40 on airplanes.
- Analysis plan was written before the run, with Wilson intervals, binomial tests against 10% chance, and Holm-adjusted McNemar pairs.
Watch out
Only Quick, Draw! categories Google already labeled correctly. Jev is text-only; SVG is still a pile of numbers in its known weak spots.
Questions about this entry
Short answers drawn from the linked project or page.
Measured, and the entry copy says the analysis plan was written before the run, with Wilson intervals, binomial tests against 10% chance, and Holm-adjusted McNemar pairs for the paired comparisons. The result on the same 400 drawings was about 35% for Jev on SVG versus about 57% for Sonnet on SVG.
Fair as a text comparison, weak as a vision claim. Both models received the same SVG path coordinates plus ten object names with one-line descriptions, so neither saw pixels. The entry copy notes Jev is text-only and that SVG is still a pile of numbers in its published weak spots.
It hides more than it breaks. The entry copy records Jev answering airplane for 209 of 400 sketches and per-object accuracy ranging from 0/40 on cats to 34/40 on airplanes, so the average is carrying a wide spread rather than a stable 35% skill.
Narrower than the headline suggests. Per the entry copy, the study uses only Quick, Draw! categories Google already labeled correctly, and no baseline beyond chance or the text-only image baseline is claimed to generalize to real images, drawing ability, or messy human input.
The drawing is squeezed into a 100 by 100 box and points that barely change the shape are dropped with the Ramer-Douglas-Peucker algorithm, then written as an SVG with one path per stroke. Jev gets that coordinate string plus ten object names, each with a one-line description such as a long thin body with two swept-back wings and a small upright tail fin.
Because TypeSafe's documentation says images, audio, and video are not supported yet, so turning each sketch into text is the only way to ask the question. It also deliberately targets a published weak spot: Jev handles words better than numbers and cannot reliably tell whether two hex colors are close, and a drawing turned into coordinates is a pile of numbers.
It lands on chance. That baseline is included alongside the Jev and Sonnet text comparisons, so the study separates a real signal from feeding the model an opaque encoding, and the article treats it as a floor rather than a fair model comparison.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
Hacker News: 400 Quick, Draw! sketches as SVG text: Jev versus Sonnet accuracy and the airplane bias.
myszon3 pointsTypeSafe's Jev Can't See. I Made It Guess What I Drew AnywayRead the thread on Hacker NewsRelated terms
Glossary definitions related to this entry.
More Jev resources like this
3GitHub stars
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPython2.5 million typed decisions recast from 670 sources: choice, score, and noul rows with per-row licenses, soft annotator targets, and Jev-audited mix weights.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
572,000 procedurally generated decision problems where labels are computed exactly from rules in the state: 13 task families, 4–60 options, and difficulty calibrated against Jev 1.13.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
0GitHub stars
Independent pre-registered audit of Jev on Spanish: 19,200 calls over 3,200 paired items on XNLI, PAWS-X, MASSIVE and Belebele. A Spanish state costs 3.0-6.4 accuracy points and roughly doubles ECE on the two hardest tasks, while Spanish instructions change nothing; a CLI reruns the same comparison on your own data.
Jev Benchmarks & Evaluations#community#evaluation#calibration
Community
marcosmartinez/jev-acentoPython
marcosmartinez/jev-acentoPython4.9kGitHub stars
LangWatch Instant Evals ships a TypeSafe Jev classifier that batches Noul, Score, and Choice judgments on production traces in one POST per text.
Jev Benchmarks & Evaluations#community#typescript#evaluation
Community
langwatch/langwatchTypeScript
langwatch/langwatchTypeScript3GitHub stars
A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
AnthusAI/Jev-CalibrationPython
AnthusAI/Jev-CalibrationPythonFrom the Jev guides
Original write-ups on what this entry is for.
Base decision checkpoints sit near chance on purpose. The typed-decisions loop, the corpora behind the students, and the temperature fit that turns probabilities into thresholds.
Jev Practices & Patterns#fine-tuning#calibration#open-models
Read article
TypeSafe publishes evals, but the numbers that matter are the ones measured on your task. A tour of the independent benchmarks, field reports, and the recurring finding that question design dominates.
Jev Practices & Patterns#benchmarks#evaluation#calibration
Read article
From the Jev community
Posts from builders shipping with Jev right now.