Jev glossary
Evaluation & TrainingProbability standard deviation
Repeatability
How stable answers are across repeated evaluations, measured separately from accuracy and reported as probability standard deviation.
- Category
- Evaluation & Training
- Also known as
- Probability standard deviation
- Reading time
- 1 min
- Related terms
- 4
- Directory entries
- 2
- Docs
- docs.typesafe.ai
- Added
- 2026-09-24
Definition
The consistency cookbooks repeat the same rubric 15 times per condition and report a mean probability standard deviation of about 0.01 for jev-1.13.0, against 0.0245 to 0.0543 for five LLM distributions.
Repeatability is not correctness: Jev still flipped its top label on two of its own questions, and a deterministic temperature-0 baseline can score perfect agreement by never abstaining. The docs keep the two measurements distinct.
Tagsevaluationconsistency
Related terms
Definitions that connect to this one.
Self-consistencyRepeatability across repeated evaluations: Jev's probability standard deviation measured at about 0.01 in the consistency cookbooks.PatternsCalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceWorkflow evalsTypeSafe's published evaluations of four automation workflows, comparing Jev and frontier LLMs as structured workflows versus single prompts.Evaluation & TrainingUncertainty bandA middle range of probabilities, such as 0.30 to 0.70, that maps to an explicit uncertain outcome instead of a hard yes or no.Answers & Confidence
From the Jev directory
Directory entries that go deeper on this term.
A repeatability study: an 8-question moderation rubric run 15 times per condition shows a mean probability standard deviation of 0.0098, and a 0.60 uncertainty gate lifts agreement to 99.2%.
Jev CookbooksDocs#official#cookbook#consistency
Official
A 14-Noul claims-triage rubric over one insurance claim, repeated 15 times: mean probability standard deviation of 0.0102 at 111 ms per call, with a 0.30 to 0.70 band for human review.
Jev CookbooksDocs#official#cookbook#consistency
Official
From the Jev AI model community
Posts from builders shipping with Jev right now.