Jev can generate game levels in real time. Faster and cheaper structured output could be a big deal for game dev!
Evaluation & Training
Workflow evals
TypeSafe's published evaluations of four automation workflows, comparing Jev and frontier LLMs as structured workflows versus single prompts.
- Category
- Evaluation & Training
- Also known as
- —
- Related terms
- 4
- Directory entries
- 48
- Docs
- evals.typesafe.ai
- Added
- 2026-09-24
Definition
evals.typesafe.ai covers security incidents, agent trace observability, invoice processing, and customer service. Models are scored as workflows of Noul, Choice, and Score questions against consensus labels from GPT-6 Astra and Claude Fable 5.1 at high thinking; Jev averaged 67.8% accuracy at $0.0004 and 0.4 seconds per case.
The headline finding is not about Jev alone: every model was more accurate, cheaper, and faster in a workflow than with the same policy as a single prompt. The evals are vendor-run, and the harness and reference labels are assumed rather than independently audited.
Tagsevaluationbenchmarks
Related terms
Definitions that connect to this one.
Consensus labelsReference answers built from two frontier models at high thinking, used as the ground truth in TypeSafe's workflow evals.Evaluation & TrainingCost/quality frontierPlotting accuracy against cost and time on log scales, where up and to the left is better; cascades are designed to sit there.Evaluation & TrainingBenchmark claimsTypeSafe's headline multipliers — 193.6x faster, 444.6x cheaper, a 238x lower input price — come from its own workflow comparisons.Evaluation & TrainingCalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & Confidence
From the directory
Published evaluations of four automation workflows, security incidents, agent trace observability, invoice processing, and customer service, comparing Jev and frontier LLMs as structured workflows versus single prompts.
Practices & PatternsDocs#official#benchmarks#evaluation
Official
The index of TypeSafe's worked examples, from parallel questions and reranking to guardrails, date extraction, and self-consistency, each with datasets and measured results.
Sites & GuidesDocs#official#docs#cookbook
Official
Scott Chacon runs hosted Jev against two local decision models, laya and kev, in a Tetris match and a GitHub Settings filter, with a public test repo.
Practices & PatternsVideo#community#video#open-models
Community
A head-to-head benchmark that prices 200 products from descriptions with Jev against GPT-5.6 Luna and GPT-4.1-Nano.
Practices & PatternsVideo#community#video#benchmarks
Community
A release guide with eval tables, pricing, curl and Python examples, and fit boundaries for Jev's early access.
Sites & GuidesArticle#community#article#pricing
Community
A heavily footnoted claims audit that separates documented Jev facts from marketing, including the seed round, third-party tests, and missing transparency.
Sites & GuidesArticle#community#article#analysis
Community
42 more matching entries in the full directory.
From the community
Posts from builders shipping with Jev right now.
Game levels generated in real time
900 images in 40 seconds
Made myself a little image classifier with OCR + Jev It was able to categorise ~900 images in 40 seconds Pretty cool
Intent-based search in Gmail
Jev is really good at intent-based search! How it looks in Gmail: (for a huge inbox you'd prob let semantic search / embeddings pull first but still much better experience)
@dabit3
Another crazy @typesafeai Jev example: Predictive spreadsheets Spreadsheets recalculate numbers, not meaning. Jev reads intent. Type "Urgency" at the top of a column and, as you type, it figures out you want each row rated from "no follow-up needed" to "urgent" in ~100 ms.
End-to-end tests run by agents
e2e + jev from @typesafeai ⚡ I'm building an open-source framework for running e2e tests with agents. supports web, mobile (and more!) available soon: tester.army/e2e
An always-on assistant with no wake word
Ai is evolving. Jev can be armed at all times. I can speak freely and it knows ( from probabilities ) if im asking my computer to do something or blaberring away at something else. no wake word. speed + affordability + intelligence is getting to the point where an always Show more
400 companies matched to one candidate
JEV is INSANE. We gave it 400 companies and one candidate profile. In 12 seconds, it predicted which jobs the candidate had the highest chance of getting, assigned a confidence score and detected job-candidate mismatches. All for just $0.0005 It can also score Show more