Jev glossary
Evaluation & Training
Brier score
A score for a full probability vector against one observed label. Lower is better, and a confident wrong answer costs more than a hedge.
- Category
- Evaluation & Training
- Also known as
- None
- Reading time
- 1 min
- Related terms
- 4
- Directory entries
- 2
- Docs
- Not listed
- Added
- 2026-09-28
Definition
The r/AmItheAsshole benchmark grades every verdict probability, from 0 for a perfect call to 2 for the worst. On 770 posts from 2025, unadjusted Jev scored 0.369 and Sonnet 5 scored 0.344. Top-1 accuracy is reported beside it and is a different number.
Rizzo Flow reports Brier on typed-decisions as well: 0.205 after its fine-tune, against 0.480 for the untuned Spark-X2.5-4B and 0.148 on Jev's dataset card. Compare Brier only when the label set and the weighting match.
Tagsevaluationcalibration
Related terms
Definitions that connect to this one.
CalibrationThe property that outcomes given a probability of 0.2 occur about 20% of the time, measured across many predictions.Answers & ConfidencePlatt scalingA two-parameter sigmoid fit on labeled data that maps a model's raw probability onto a better-calibrated one.Evaluation & TrainingBenchmark claimsTypeSafe's headline multipliers (193.6x faster, 444.6x cheaper, a 238x lower input price) come from its own workflow comparisons.Evaluation & TrainingProbability distributionThe floats summing to 1 across your options or levels that every Choice and Score answer returns under probabilities.Answers & Confidence
From the Jev directory
Directory entries that go deeper on this term.
A 770-post benchmark that scores Jev, Sonnet 5, GPT-5 nano, and two local models on Reddit verdicts with weighted Brier scores.
Jev Benchmarks & Evaluations#community#python#evaluation
Community
dchristopoulos/jev-aitaPython
dchristopoulos/jev-aitaPythonA local server that reads next-token probabilities from a fine-tuned Spark model and returns Choice, Score, and Noul answers with zero generated tokens.
Jev Models & Reimplementations#community#python#open-models
Community
Rizzo-AI-Academy/rizzo-flowPython
Rizzo-AI-Academy/rizzo-flowPythonFrom the Jev AI model community
Posts from builders shipping with Jev right now.