Jev by @typesafeai is now on OpenRouter, in beta. Jev is a System One model. Instead of generating text, it takes your app's state plus a typed question and returns a typed decision with a probability attached. There is no JSON prompting, parsing layer, and nothing to validate Show more
Jev as a judge for agent evals
An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
- Category
- Practices & Patterns
- Published by
- Community
- Author
- danielgshea
- Added
- 2026-09-22
Tagscommunitypythonevaluationbenchmarkscalibration
Highlights
- Across 500 repeated does_pass decisions per judge, Jev matched every human label for 100.0% accuracy versus 99.8% Terra, 96.4% Luna, and 80.0% Sonnet.
- Jev's mean per-case quality variance was 0.0000149; Luna was 433x higher, Terra 913x higher, and Claude 92x higher.
- Five cases (Seattle, Austin, Dublin, Tokyo, Springfield) were captured once and reused so only the judges introduced variation.
- The README calls this a small corpus with one human reviewer and states it is not a general ranking of judge accuracy.
Quickstart
bash
cp .env.example .env
# Add your API keys to .envWatch out
No license file, so reuse terms are unclear. Needs Python 3.13+, plus TypeSafe, Tavily, and workspace-scoped LangSmith API keys; the results come from five runs and one human reviewer.
More like this
74GitHub stars
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Practices & Patterns#community#python#benchmarks
Communityjevbench
17GitHub stars
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Practices & Patterns#community#python#benchmarks
Communityjev-benchmarks
5GitHub stars
An independent calibration study of Jev over three public benchmarks and 900 rule-generated support tickets, publishing every raw Gateway response and the quantization limits of returned probabilities.
Practices & Patterns#community#python#calibration
Communityjev-ood-calibration
From the community
Posts from builders shipping with Jev right now.
Jev lands on OpenRouter
700 leads scored for $0.09
JEV is INSANE. We gave it 700 high-intent leads and personalised outreach messages. In 40 seconds, it predicted how each message would perform, assigned a confidence score and detected lead-message mismatches. All for just $0.09. JEV can also score leads, analyse buying Show more
Beating Gemini Flash Lite on an eval
Ran @typesafeai's Jev against an existing classifier eval that previously used Gemini 2.5 Flash Lite. It won both on quality (saturated the eval) and speed (6x)
Browser Use Ultrafast, powered by Jev
Breaking: Browser Use + Jev = Ultrafast ⚡ Findings flights took 7s and cost only $0.0039 🤯 > new action space every step > DOM state space > small LLM fallback to type (this video is at 1x speed btw) Built a tiny open source browser agent. try it below ↓
A really smart switch statement
hype-free explanation of jev: jev does not replace gpt / claude jev is just a *really* smart switch statement like if 2016 ml classifiers got 2026 levels of intelligence it's a new* type of tool that will make a lot of workloads insanely fast, cheap, and accurate * = and by Show more
@CompleteSkeptic
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x
When a designer gets Jev
when a designer gets access to Jev