Jev glossary
Patterns
Self-consistency
Repeatability across repeated evaluations: Jev's probability standard deviation measured at about 0.01 in the consistency cookbooks.
- Category
- Patterns
- Also known as
- None
- Reading time
- 1 min
- Related terms
- 4
- Directory entries
- 2
- Docs
- docs.typesafe.ai
- Added
- 2026-09-24
Definition
The Choice cookbook runs an 8-question moderation rubric 15 times per condition and measures a mean probability standard deviation of 0.0098, against 0.0245 to 0.0543 for five LLM distributions. The Noul cookbook measures 0.0102 on a 14-question claims rubric.
These are repeatability numbers, not accuracy numbers: Jev still flipped its top label on two questions across its own repeats, and a deterministic temperature-0 baseline can score perfect agreement by never abstaining.
Tagspatternsconsistencyevaluation
Related terms
Definitions that connect to this one.
RepeatabilityHow stable answers are across repeated evaluations, measured separately from accuracy and reported as probability standard deviation.Evaluation & TrainingUncertainty bandA middle range of probabilities, such as 0.30 to 0.70, that maps to an explicit uncertain outcome instead of a hard yes or no.Answers & ConfidenceCalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidenceWorkflow evalsTypeSafe's published evaluations of four automation workflows, comparing Jev and frontier LLMs as structured workflows versus single prompts.Evaluation & Training
From the Jev directory
Directory entries that go deeper on this term.
A repeatability study: an 8-question moderation rubric run 15 times per condition shows a mean probability standard deviation of 0.0098, and a 0.60 uncertainty gate lifts agreement to 99.2%.
Jev CookbooksDocs#official#cookbook#consistency
Official
A 14-Noul claims-triage rubric over one insurance claim, repeated 15 times: mean probability standard deviation of 0.0102 at 111 ms per call, with a 0.30 to 0.70 band for human review.
Jev CookbooksDocs#official#cookbook#consistency
Official
From the Jev AI model community
Posts from builders shipping with Jev right now.