Skip to content
JevDirectory.orgTraffic count loading. View site trafficSponsor Us
CommunityBenchmarks & Evaluations

Jev calibration study

A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.

Open github.comAnthusAI/Jev-Calibration
Category
Benchmarks & Evaluations
Format
—
Published by
Community
Author
AnthusAI
Added
2026-09-28
Last verified
2026-09-28

AnthusAI/Jev-Calibration

Organization repository on GitHub

View on GitHub

Does Jev's confidence mean what it says? Calibrating Jev (TypeSafe System One) with Platt scaling and isotonic regression.

GitHub stars
3
Forks
2
Primary language
Python
Last pushed
Updated Sep 2026

Repo stats from the GitHub API, cached Sep 2026.

Tagscommunitypythonevaluationcalibration

Highlights

  • On 8,801 unique sentiment examples, raw expected calibration error for P(positive) was 0.117.
  • Platt scaling cut that to 0.052. Isotonic regression cut it to 0.008 on the held-out split.
  • Isotonic matched or beat Platt at every calibration size from 20 to 5,280 examples.
  • Noul accuracy rose from 47.6% at 50–60% confidence to 100% above 95% (2,104 of 2,104).
  • For a two-option Choice, the confidence field matched 2 times p_top minus 1, within rounding.

Quickstart

bash
# results live in results/variants.json and results/raw_confidence_bands.json

Watch out

No license file. One constructed sentiment set and jev-1.13.0. Neutral-tier labels are arbitrary by design. The whole dataset was about 330 input tokens per request and a median latency of 0.24 seconds.

Related terms

Glossary definitions related to this entry.

More like this

Jev on r/AmItheAsshole
3GitHub stars
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Benchmarks & Evaluations#community#python#evaluation
Communitydchristopoulos/jev-aitaPython
JevBench
150GitHub stars
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Practices & Patterns#community#python#benchmarks
Communityfstandhartinger/jevbenchPython
Jev as a judge for agent evals
93GitHub stars
An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Practices & Patterns#community#python#evaluation
Communitydanielgshea/jev-as-a-judgePython
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Navigating Neo4j with Jev

思维怪怪
思维怪怪
@0xLogicrw
·

Jev 这个 waitlist 还是很给力的，昨天申请，今天就能用上。 给已经拿到 API、但还不知道怎么玩的人整理了一份 Awesome Jev，目前我能确认到的 Jev 项目基本都在这里： 1. jev-ultrafast Browser Use 做的高速浏览器 Agent。Jev Show more

Image
思维怪怪
思维怪怪
@0xLogicrw

前 OpenAI 研究员 Diogo Almeida 创办的 TypeSafe AI 推出新模型 Jev。它有点像一个能读懂自然语言的超级分类器，不生成文本，只返回选项、分数和概率，专门给软件做判断。 普通大模型需要一个 token 一个 token 往外生成，Jev 则可以并行给出多个结果。TypeSafe 还用新的 RLCD

Reply
Read 35 replies

Reranking 33,047 catalog entries

Jason Zhu
Jason Zhu
@GoSailGlobal
·

拿 Jev 做搜索重排，我先泼一盆冷水：单独用，它没打赢向量检索 TypeSafe 的 Jev 这阵子很火，一堆项目拿它做重排。我们在 Agent Skills Hub 的 33,047 条目录上认真测了一次，164 条中英文真实查询，9,831 对分级标注，整套只花了 2.6 美元 三个结论 01｜单独重排，约等于没赢 Jev 重排 bge-m3 Show more

Watch on X
Jason Zhu
Jason Zhu
@GoSailGlobal

有美团、阿里的老哥嘛？ 试试加一路召回、重排（离线、近实时实现），我觉得有奇效 他在文本理解上 跟之前机器学习、llm很不一样 还能自动打标签做特征

Reply
Read 6 replies

Six uses that stuck after 60 days

Security decisions that fit Jev

Kostas
Kostas
@Kostastsale
·

This made me rethink where AI actually fits into security engineering. For purely engineering work, forget about ChatGPT or Claude. TypeSafe AI just released Jev, and I think it’s going to change how we build AI into security workflows. Instead of asking an LLM to “investigate Show more

TypeSafe AI
TypeSafe AI
@typesafeai

we are officially out of stealth! join the frontier and get access to Jev on our website (link on profile)

Reply
Read 15 replies

A million judged questions

Inferring Jev's internals from 1,000 calls

iwashi / Yoshimasa Iwase
iwashi / Yoshimasa Iwase
@iwashi86
·

Jevの内部アーキテクチャを推測している技術記事(Jev’s Architecture Unmasked)からメモ。 ・本記事はJevのAPIを約1万回の呼び出して、内部構造を推測したもの ・従来の言語モデルを用いた分類やルーティングでは、トークンを1文字ずつ逐次生成するために膨大な無駄な計算コストが発生していた。 Show more

Reply
Read 2 replies