Is "the probabilities are not calibrated" just noise from a small sample?

The repo recomputes each study's ECE against the noise floor implied by its own sample size and binning, then runs its own 800-item gradient across four difficulty tiers. ECE sits at 2.1-2.5x that floor in every tier, including the one where accuracy reaches 97.5%, so the gap is not a small-sample artifact.

So does this mean Jev is unusable as a classifier?

The conclusion is narrower: Jev returns a well-behaved monotone score carrying a stable distortion rather than a probability. Because the error is underconfident at the top here, thresholding at p >= 0.9 hit 1.000 in every tier at 21.5-32.5% coverage, so the sign of the error decides whether a threshold is safe, not the size of the ECE.

How many labels before correcting the probabilities is actually worth it?

Fifty is described as a floor rather than a recommendation. The Platt slope stays between 2.11 and 2.61 across tiers, so transferring it and refitting only the intercept on 50 labels cut held-out ECE by 62%. Below 30 labels the correction can leave calibration four times worse than doing nothing.

Is this a repository I can run, or just an essay?

There is an analysis/ directory with scripts for the noise floors, calibration transfer, set size, quantisation and circularity, a lab/ tree with committed raw per-call JSONL and labelled datasets, and a pytest suite that runs entirely offline with no API key or network. Only a handful of the listed commands need a key.

What are the API limits I should plan around?

255 Choice options and about 32,000 input tokens per request. Errors are 401 for a bad key, 422 for a malformed question, and 429 or 529 requiring backoff, and the repo notes there are no published rate limits, SLA, or uptime commitment. Pricing is $0.042 per million input tokens with output free.

What did it find about the returned number resolution?

The probabilities are quantised to 0.01, and Choice and Score can return exactly 0 or 1, which nothing in a direct response announces. An exact 0 on the correct option cannot be repaired by any rescaling. Noul did not return an exact 0 or 1 once in 11,148 answers, and every calibration result in the repo is Noul.

Does batching many questions change the answers?

It found batching is near-free at about 0.33 ms per extra question on roughly 1.4 s of fixed overhead, and that questions cannot see each other: a code word placed in one question's instructions scored 0.02 on a sibling question against 0.99 in the shared state.

CommunityJev Practices & Patterns

Jev Exploration

A claim ledger for Jev that recomputes published calibration figures against their own noise floors and runs an 800-item difficulty gradient on jev-1.13.0.

See highlights

Open github.com

Stars: 3
Language: Python
License: MIT
Published by: Community
Author: SamuelSacco
Last verified: Oct 2026

SamuelSacco/jev-exploration

User repository on GitHub

View on GitHub

Jev (TypeSafe) exploratory thread: claim audit, live demos, and runnable code

Forks: 0
Last pushed: Updated Oct 2026

Repo stats from the GitHub API, cached Oct 2026.

About this project

What is actually known about TypeSafe's Jev, the "System One" judgment API that returns typed probabilities instead of generated text. Started as a deep dive during launch week (September 2026), now maintained as a ledger of claims and the evidence behind them, including other people's benchmarks. → The ledger: every claim, its status, and its source.

Quoted from the project's README. Read the full README.

Tagscommunity jevbest catalog-import awesomejev

Works forJev for Classification

Used inJev for Software & DevTools

Languages: Python

Highlights

ECE lands 2.1-2.5x its noise floor across four difficulty tiers, including the tier where accuracy is 97.5%.
The distortion is compression toward the middle, crossing over near 0.5: low probabilities overstated, high ones understated.
p >= 0.9 gave a 1.000 hit rate in every tier at 21.5-32.5% coverage, but 73.9% on jev-phishing-bench, which errs the other way.
Transferring a fitted Platt slope and refitting only the intercept on 50 labels cuts held-out ECE by 62%.

Watch out

Every result is Noul on one version, jev-1.13.0, and the probabilities are quantised to 0.01 with Choice and Score able to return exactly 0 or 1, which no rescaling can repair. A perfectly calibrated model already scores ECE around 0.045 at the n=60 sample size and binning a widely cited benchmark uses, so that figure cannot measure calibration in either direction. Whether the slope survives a domain change or a retrain is untested. The CI definition under ci/ is not enabled yet. Copy reviewed against the repository README on 2026-10-04, not from running it.

Questions about this entry

Short answers drawn from the linked project or page.

More Jev resources like this

jev-on-a-laptop

Unofficial laptop study of parallel constrained decoding: prefill once, share the KV cache across fields, and decide all schema fields in one batched forward pass without writing JSON.