Jev glossary
Evaluation & TrainingError floor
Oracle floor
The best error rate achievable with perfect choices from the available options, used to judge how close a system gets.
- Category
- Evaluation & Training
- Also known as
- Error floor
- Reading time
- 1 min
- Related terms
- 3
- Directory entries
- 1
- Docs
- docs.typesafe.ai
- Added
- 2026-09-24
Definition
The skill suggestion cookbook measures wrong skill loads at 16.8% before and 7.3% after, against an oracle floor of 2.5%, and needless loads at 9.8% to 4.0% against a floor of 1.2%. The floor is what a perfect chooser would score given the same catalog and inputs.
Floors make improvement legible: halving the gap to the oracle is progress, while a raw percentage alone cannot say whether a system is near the ceiling.
Tagsevaluationbenchmarks
Related terms
Definitions that connect to this one.
Skill suggestionRank a large skill catalog in one request, then re-read the top few in full detail and pick at most one.PatternsRepeatabilityHow stable answers are across repeated evaluations, measured separately from accuracy and reported as probability standard deviation.Evaluation & TrainingWorkflow evalsTypeSafe's published evaluations of four automation workflows, comparing Jev and frontier LLMs as structured workflows versus single prompts.Evaluation & Training
From the Jev directory
Directory entries that go deeper on this term.
Rank 182 agent skills in one request and re-read the top three in a second: over 488 requests, wrong skill loads fell from 16.8% to 7.3% and needless loads from 9.8% to 4.0%.
Jev CookbooksDocs#official#cookbook#agents
Official
From the Jev AI model community
Posts from builders shipping with Jev right now.