Jev and System One Models: Calibration Beats Accuracy

Not in this thread. The objection is that when the correct answer is not among the options, there is no way to signal absence. The suggested fix is to always declare an explicit none-of-the-above option and branch outside the model.

One commenter tested exactly that, feeding irrelevant state and asking questions about sporting outcomes, and got nothing near the obviously correct no-information answers nor near 0.5 when the prior sat far from it. Stating the prior in the state barely helped.

Of your dataset. The thread's repeated point is that calibration has to be measured where you deploy it, and probabilities that sum to one do not make a model calibrated. One commenter points at ad click prediction, where calibration is the entire game.

No. The entry notes he had no API access at the time, so the performance numbers in the piece are labeled as the vendor's claims rather than his own. That matters when weighing the argument, alongside the thread's spam accusations about young accounts.

Ask several specific yes/no questions and feed the probabilities into a logistic regression rather than asking one score over ordered levels. The argument is that a narrowly phrased question like is the customer angry would agree across models and with human raters, while is it urgent is much harder to answer directly, so the narrower decomposition stays stable as models change.

In principle, yes, at high cost and probably poor quality. One comment points out you could run it as an autoregressive loop asking it to predict the next character, sample at a temperature, and repeat, and even guesses that correctly guessing the tokenizer and requesting a choice over next tokens would improve it. He notes he is not inclined to try it.

Poorly. One reply calls it super annoying and content free, and the thread carries spam accusations about the poster's account age alongside a YC Winter 2027 joke. The substance of the disagreement is still worth reading, but the reception is not a good-faith evaluation of the author's argument.