Skip to content
JevDirectory.org
CommunityPractices & Patterns5 starsVerified 2026-09-22

Does Jev know when it doesn't know?

An independent calibration study of Jev over three public benchmarks and 900 rule-generated support tickets, publishing every raw Gateway response and the quantization limits of returned probabilities.

Category
Practices & Patterns
Published by
Community
Author
scienthoon
Added
2026-09-22
Tagscommunitypythoncalibrationevaluationbenchmarksconfidence

Highlights

  • 3,721 public-benchmark items and 900 synthetic tickets, all through the Vercel AI Gateway with zeroDataRetention, for about $0.06 in API calls.
  • On public sets the probabilities look in-domain calibrated: OpenBookQA 94.2% accuracy at ECE 0.024, CommonsenseQA 88.1%, HellaSwag 86.1%.
  • On the unseen priority rule accuracy was 44.7% with ECE 0.325; after an endpoint-floor correction the score refit T is 1.92, still overconfident.
  • The boolean question goes the other way at refit T 0.66, so the sign of the calibration error flips by question type.
  • Probabilities are quantised to 0.01 and often exactly 0 or 1: 1,051 of 2,000 OpenBookQA option probabilities are exactly 0.

Quickstart

bash
python scripts/endpoint_refit.py --dump results/jev_synth.jsonl

Watch out

MIT-licensed code with CC0 synthetic data; one synthetic task family at n=900, the Gateway exposes no model version, and all numbers are from 2026-09-19.

More like this

74GitHub stars
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Practices & Patterns#community#python#benchmarks
Communityjevbench
An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Practices & Patterns#community#python#evaluation
17GitHub stars
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Practices & Patterns#community#python#benchmarks
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Screening agent actions with Jev

Tested TypeSafe’s Jev (no-text, probability-only model) as an AI agent safety monitor. Checking each action first worked well caught most attacks with almost no false blocks, and much faster than Gemini.

Image
Image
Image
Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

Cua's small System One models

A 706K-parameter form filler

cua open sourced a 706k param model that fills a whole form in one 50ms pass the llm agent doing the same form took 23 turns and 39.6 seconds the specialists are going to eat the generalists from the bottom

Cua
Cua
@trycua

1/ Introducing CUA-S1: a family of System One Models, small, specialized, and built for computer use. Today we're open-sourcing CUA-S1-FORMS, the first in the family: github.com/trycua/cua

Image
Reply

Navigating Neo4j with Jev

Jev 这个 waitlist 还是很给力的,昨天申请,今天就能用上。 给已经拿到 API、但还不知道怎么玩的人整理了一份 Awesome Jev,目前我能确认到的 Jev 项目基本都在这里: 1. jev-ultrafast Browser Use 做的高速浏览器 Agent。Jev Show more

Image
思维怪怪
思维怪怪
@0xLogicrw

前 OpenAI 研究员 Diogo Almeida 创办的 TypeSafe AI 推出新模型 Jev。它有点像一个能读懂自然语言的超级分类器,不生成文本,只返回选项、分数和概率,专门给软件做判断。 普通大模型需要一个 token 一个 token 往外生成,Jev 则可以并行给出多个结果。TypeSafe 还用新的 RLCD

Reply

Reranking 33,047 catalog entries

拿 Jev 做搜索重排,我先泼一盆冷水:单独用,它没打赢向量检索 TypeSafe 的 Jev 这阵子很火,一堆项目拿它做重排。我们在 Agent Skills Hub 的 33,047 条目录上认真测了一次,164 条中英文真实查询,9,831 对分级标注,整套只花了 2.6 美元 三个结论 01|单独重排,约等于没赢 Jev 重排 bge-m3 Show more

Jason Zhu
Jason Zhu
@GoSailGlobal

有美团、阿里的老哥嘛? 试试加一路召回、重排(离线、近实时实现),我觉得有奇效 他在文本理解上 跟之前机器学习、llm很不一样 还能自动打标签做特征

Reply

Six uses that stuck after 60 days