Jev glossary
Answers & ConfidenceProbability calibration
Calibration
The property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.
- Category
- Answers & Confidence
- Also known as
- Probability calibration
- Reading time
- 1 min
- Related terms
- 4
- Directory entries
- 345
- Docs
- docs.typesafe.ai
- Added
- 2026-09-24
Definition
RLCD trains Jev so higher probability corresponds to a greater chance of being correct. Across many predictions, outcomes assigned 0.2 should occur about 20% of the time and outcomes assigned 0.8 about 80% of the time.
Calibration describes groups of predictions, not any single answer. A well-calibrated 0.9 can still be wrong, which is why the docs keep a review path for individual cases that matter.
Tagsconfidencetrainingevaluation
Related terms
Definitions that connect to this one.
RLCDTypeSafe's post-training method: reinforcement learning that optimizes for calibrated decisions and probabilities instead of generated text.Evaluation & TrainingConfidenceA 0-1 number on every Choice and Score answer that collapses the probability distribution into one certainty score.Answers & ConfidenceProbability distributionThe floats summing to 1 across your options or levels that every Choice and Score answer returns under probabilities.Answers & ConfidenceWorkflow evalsTypeSafe's published evaluations of four automation workflows, comparing Jev and frontier LLMs as structured workflows versus single prompts.Evaluation & Training
From the Jev directory
Directory entries that go deeper on this term.
Why TypeSafe trains decision models with RLCD instead of RLHF: calibrated probabilities where 0.2 outcomes happen about 20% of the time, and the case for machine-to-machine automation.
Jev Guides & ArticlesDocs#official#docs#evaluation
Official
How TypeSafe reports certainty, how it differs from probability, and how to use it architecturally to gate and route decisions.
Jev Practices & PatternsDocs#official#confidence#routing
Official
arXiv paper evaluating the option probabilities Jev returns against linked responses to 20 binary items from five archived survey waves, comparing population-only prediction, prediction given recorded answers, and sequential completion against survey records.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#calibration
Community
Python project that measures how much CUAD license-agreement text Jev can bin before an expensive model reads it, reporting bin rates, lost lawyer-marked provisions, and estimated cost savings against a Claude Sonnet baseline.
Jev Benchmarks & Evaluations#evaluation#calibration#benchmark
arXiv paper testing whether Jev, a decision model that outputs judgment probabilities, works as a multilingual judge. It compares Jev with generative LLM judges across four annotated benchmarks, three evaluation formats, and up to 47 language varieties.
Jev Benchmarks & EvaluationsArticle#arxiv#evaluation#benchmark
Community
SO-Arena is a benchmark for System One models covering 11 domains and five capability families. It measures capability spectrum and decision reliability across 21 models, including decision stability under option-set changes and missing-answer recognition.
Jev Benchmarks & EvaluationsArticle#arxiv#benchmark#evaluation
Community
339 more matching entries in the full directory.
From the Jev AI model community
Posts from builders shipping with Jev right now.