Question 1

So is Jev's confidence trustworthy, or just well presented?

Accepted Answer

Mixed, and the sign depends on the question type. On three public sets the probabilities needed almost no correction, with OpenBookQA at ECE 0.024 against a 0.024 noise floor. On a rule the model cannot know, the score question sat at refit T 1.92, meaning overconfident, while the boolean question on the same inputs went the other way at T 0.66. Read the direction, not the magnitude.

Question 2

Are these benchmark numbers contaminated, and does the author admit it?

Accepted Answer

The author explicitly says contamination cannot be verified either way, and labels the public-set calibration in-domain. The reported accuracies sit far above what a 3-8B model scores zero-shot on the same splits, which is consistent with those train splits being in the training mix. The Gateway exposes no model version, so nothing in the run pins the weights behind the numbers.

Question 3

Elsewhere people report no misses at confidence 1.000 — is that safe to rely on?

Accepted Answer

Not on this evidence. The study reports a direct counterexample: one OpenBookQA item assigned probability 0.00 to the correct answer and 1.00 to a wrong one, at confidence 0.99. Probabilities are quantised to 0.01 and endpoints are used freely, with 1,051 of 2,000 option probabilities in that set exactly 0, and a temperature cannot repair an exact zero.

Question 4

Should I threshold on the separate confidence field instead of the max probability?

Accepted Answer

No. On these sets that field was never better than the max probability and sometimes much worse: its ECE was 0.035 on OpenBookQA, 0.078 on HellaSwag, and 0.18 on the synthetic set. The advice the repo gives is to treat the output as a monotone score and calibrate per question rather than per model, since the sign of the error flips by question type.

Question 5

How was this measured, and what would it cost me to rerun it?

Accepted Answer

Every call went through the Vercel AI Gateway with model id typesafe-ai/jev and zeroDataRetention enabled, driven by AI SDK 7.0.107 experimental_evaluate, with 3,721 public-benchmark items and 900 synthetic tickets for about $0.06 and zero failed calls. The typesafe-ai/jev-latest id in the AI SDK docs returns Model not found on the Gateway.

Question 6

How bad was the task the model could not know, in plain numbers?

Accepted Answer

On the priority rule, which is defined outside the ticket text so no zero-shot model can recover it, accuracy was 44.7% against 25% chance and ECE was 0.325. The level Jev picked still carried 0.74 probability on average, which is the part that matters for routing: the number does not fall back toward the base rate when the model is guessing.

Question 7

How does this compare with the other independent calibration studies?

Accepted Answer

It is not the first, and the repo links the ledger. jev-spam-eval reports 98.6% on about 9,900 emails but no ECE, and jev-phishing-bench reports 62.6% at ECE 0.154 on 2,000 emails against Claude Haiku 4.5's 0.097. Those studies use boolean questions on labels recoverable from the text; this one adds an unknowable label and the sign-by-question-type reading.

Does Jev know when it doesn't know?

About this project

Highlights

Quickstart

Questions about this entry

From the Jev guides

From the Jev AI model community