Skip to content
Evaluation & Training

Isotonic regression

A non-decreasing fit from raw confidence to observed accuracy. It can follow a miscalibration that is not a sigmoid.

Category
Evaluation & Training
Also known as
—
Related terms
4
Directory entries
1
Docs
—
Added
2026-09-28

Definition

The sentiment study fit isotonic regression on a calibration split and measured expected calibration error of 0.008 on the held-out test, against 0.117 raw and 0.052 for Platt. It matched or beat Platt at every calibration size from 20 examples to 5,280.

After that fit, question wording moved raw error from 0.064 to 0.160 across 11 setups, and the calibrated error stayed between 0.006 and 0.018. The fit is for one dataset and jev-1.13.0. A new label set needs its own.

Tagscalibrationevaluation

From the directory

A sentiment study of whether Jev's stated confidence matches accuracy, comparing raw scores with Platt scaling and isotonic regression.
Benchmarks & Evaluations#community#python#evaluation

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

jev(): Postgres WHERE clauses in plain language

I think I just cooked something 🔥 jev(): a PostgreSQL extension that searches your whole database in natural language. No index, no embeddings, just one function. WHERE jev(people, 'could work from home') or WHERE jev(people, 'name sounds european') 129 rows judged in ~1s  Show more

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

A real-time ad blocker that classifies every DOM element

I build an undetectable realtime adblocker extension with typesafe It checks every dom element and classifies as ad/non-ad and removes it if true Extremely fun to work with, expecting an incredible shift in how AI is being used in the future

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

Jev plus Astra beats the Ender Dragon for under a dollar

Tool calling as classify plus action, back to twelve-factor agents

jev is the best excuse you could possibly have to go re-read 12 factor agents. Tool calling itself can be decomposed into classify+action, if you learn to design ai programs as pipelines that switch breathlessly between classification, structuring data, deterministic code, AND Show more

Dillon Mulroy
Dillon Mulroy
Cloudflare
@dillon_mulroy

i think jev is resonating with devs so well b/c it unlocks so many opportunities for composing ai into systems and products rather than ai _becoming_ the product/system really does feel like it was a missing primitive

Reply

Why Jev encodes a state and outputs distributions instead

Jev is one of the more interesting model launches I have seen recently because it asks a very simple question: Why are we using autoregressive LLMs as insanely expensive if statements inside software? The easiest way to think about it is: LLM: text -> generate tokens Show more

Image
Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

How the agent stack moved in Jev's first three days