An early-access test of TypeSafe's Jev: calibrated judgments for half a cent

The measurable part is independent: the author ran the model on 24 documents, published the agreement, calibration, cost and latency figures, and notes TypeSafe had not seen the post or the numbers beforehand. The training story is not checkable, though. RLCD is presented as the alternative to RLHF, but no paper is published, so its novelty cannot be judged from outside.

It handled Norwegian well enough to be useful. The trial ran 24 hearing responses in bokmål and nynorsk, from three-sentence web-form replies to long legal commentary and council minutes, and returned typed answers with probabilities for all of them. The caveat is the docs' own claim that Jev excels in English with other languages varying, plus a practical limit worth computing in your own language.

Directionally right on this trial. Across 192 argument judgments the 0.0 to 0.1 bin held 0% yes and the 0.9 to 1.0 bin held 98%, which is what you want if you plan to threshold on them, and the author calls it slightly underconfident. Two caveats: reference labels come from Claude Fable 5.1, so this measures agreement with a frontier model, and 24 documents is a first look.

It hurt here, which is the counter-intuitive result. Rewriting short draft questions with careful qualifiers dropped agreement with the reference labels from 0.89 to 0.86 and pushed calibration error from 0.040 up to 0.116. If you are used to prompt-tuning a classifier, measure your wording rather than assuming more precision helps. Separately, state caps at 32k tokens, about 64,000 characters of Norwegian at 2.06 characters per token.

24 Norwegian responses to the 2022 hearing on a resource rent tax for salmon farming, drawn from 412 published responses that include bokmål and nynorsk, legal PDFs and council minutes.

The letter's stance came back with probabilities (against 0.62, for_with_changes 0.37, no_clear_stance 0.01) and the reference label was for_with_changes, so Jev got that one wrong but reported low confidence.

RLCD is reinforcement learning for calibrated decisions, aiming at a probability that holds up when counted, while RLHF aims at the response a person likes best and can be compelling without being reliable for unattended automation.