Jev can't be calibrated — alexmolas.com

Calibration is a property of a model and a data distribution together, not of a model alone. Jev returns the same probability for the same input and prompt regardless of who asks, so a fit on TypeSafe's data can be miscalibrated on your traffic even if the training was sound.

It is an anecdote, linked from tweets he found while writing rather than a measurement in the post. What makes it more than drift is that the true probability was stated in the prompt and the model still answered 0.92, so it is not explained away by a different data distribution.

Apparently not. A linked experiment finds Noul much better calibrated than Choice on the same problem, which is awkward because it leaves the meaning of calibrated probabilities dependent on which primitive you used.

Useful, in his view. It classifies without a training set, which a fine-tuned BERT cannot, and it is a universal classifier rather than one tied to a single task. His prescription is to treat outputs as good scores and fit Platt scaling on a few hundred of your own labeled examples before trusting the number.

The vendor's. TypeSafe describes Jev as its first System One Model: it takes unstructured input and returns typed decisions from a set of outputs you define in advance, each with a probability attached. The selling point is that all answers come with calibrated probabilities and confidence scores, trained with RLCD, reinforcement learning for calibrated decisions. The post takes on that specific claim, not on Jev's usefulness.

Whenever your system depends on the number rather than the order. Ranking survives, because the scores do order examples well. Thresholds, expected-cost calculations, and combining Jev with another model do not, since each of those assumes the probability means something about real frequency. His advice is to measure calibration on your own data before trusting any of them.

A 617-word, three-minute post dated 23 September 2026 that presents no measurement of its own. The fair-coin 0.92 comes from tweets he found while writing, and the Noul-versus-Choice finding comes from a separate recent experiment; both are linked rather than reproduced. Background on Jev is Simon Willison's overview. Read it as a position argument, not as new evidence.