Docs
OfficialJev Cookbooks
Cookbook: Skill Suggestion
Rank 182 agent skills in one request and re-read the top three in a second: over 488 requests, wrong skill loads fell from 16.8% to 7.3% and needless loads from 9.8% to 4.0%.
- Format
- Docs
- Reading time
- 3 min
- Published by
- TypeSafe AI
- Last verified
- Sep 2026
Used inJev for Software & DevTools
Highlights
- 182 skills across 33 categories, 488 requests, with 315 covered by 171 distinct skills and 173 uncovered.
- Wrong loads dropped from 16.8% to 7.3% and needless loads from 9.8% to 4.0%, against oracle floors of 2.5% and 1.2%.
- Call one runs a Choice over all 182 skills plus three gate Nouls; call two re-reads the top three in full detail.
- The winner is appended as one system-prompt line after the roster so the prefix cache still holds.
Quickstart
python
response = client.system_one(
state=build_state(request),
questions={
"which": Choice(instructions=CHOICE_INSTRUCTIONS,
criteria={s["name"]: s["description"] for s in ROSTER}), # 182 skills
"gate::acts_on_user_system": Noul(instructions=GATE_QUESTIONS["acts_on_user_system"]),
},
model=TYPESAFE_MODEL,
)
ranked = sorted(response.answers["which"].probabilities.items(), key=lambda kv: -kv[1])
shortlist = tuple(name for name, _ in ranked[:SHORTLIST]) # top 3 -> second request
# GATE_THRESHOLD = 0.30: mean gate below this, suggest nothingWatch out
It fixed 37 covered requests but broke 7 the agent had right, and a confident wrong suggestion is persuasive.
Questions about this entry
Short answers drawn from the linked project or page.
The first ranks all 182 skills and runs the gate Nouls; the second re-reads the top three in full detail. The winner is appended as one system-prompt line so the prefix cache still holds.
Yes. The caveat says it fixed 37 covered requests but broke 7 the agent had right, and a confident wrong suggestion is persuasive — wrong loads still fell overall from 16.8% to 7.3%.
That is the problem. An agent with a large roster makes its choice on almost no information, because the roster reaches it as one truncated line per skill; Hermes cuts descriptions to 60 characters by default, at which width the skill that edits .pptx files reads nearly the same as the one that authors them. Loading everything also inflates cost and degrades selection.
Over 488 requests against claude-haiku-4-5-20251001 with the Hermes roster: wrong skill loads fell from 16.8% to 7.3% and needless loads from 9.8% to 4.0%, against oracle floors of 2.5% and 1.2%. That floor is not zero, because even an agent handed the right answer sometimes ignores it.
One extra line in the system prompt for that turn, a skill_relevance block naming one skill and telling it to ignore the block if it does not fit what the user actually asked for. The agent keeps its own index and its own judgement, and since the roster itself never changes, prefix caching over it still holds.
A pinned snapshot of 182 skills across 33 categories from Nous Research's hermes-agent, which is MIT. Call one runs a Choice over all 182 plus gate Nouls; call two re-reads the top three using 700 SKILL.md characters each against a roster file that stores 1600. The gate threshold is 0.30 on the mean Noul, below which nothing is suggested, and a shortlist whose best fit is under 0.30 is dropped.
pip install anthropic matplotlib ipython 'cooksafe>=0.2.0,<0.3.0', then export TYPESAFE_API_KEY and ANTHROPIC_API_KEY for the agent being measured. JsonCache replays results on their inputs so re-running costs nothing until you delete json_cache.json. The published run used jev-1.12 and claude-haiku-4-5-20251001, rendered 2026-07-31.
Reactions & coverage
Posts, threads, and videos about this entry from around the web.
X: Search and tagging on keep.md
X: Jev reranks station suggestions in TrainLCD
YouTube: Jev in the reranking step of a RAG pipeline: why cosine similarity fails and how a steerable reranker compares. A Colab notebook comes with it.
More Jev resources like this
Turn natural language into typed function calls: closed-set arguments become Choice questions, optional ones become Noul questions, and 54 questions ship in one request for a trading assistant.
Jev CookbooksDocs#official#cookbook#tool-calling
Official
Score retrieved passages with four Noul questions each, then route in code: prompt injections dropped, false premises kept as conflicts, and evidence assembled separately for the generator.
Jev CookbooksDocs#official#cookbook#search
Official
A support-ticket cookbook that lets Jev decide which requests are worth remembering, then wraps each call so saved tickets ride along in the state. Six later messages route correctly with memory against two without.
Jev CookbooksDocs#official#cookbook#memory
Official
Semantic search over GitHub's Terms of Service: one request ranks all 218 lines with a Choice while a Noul checks whether the document contains an answer at all, including when it should say no.
Jev CookbooksDocs#official#cookbook#search
Official
A two-stage extraction cascade: a mini model extracts, a Noul battery verifies each field in one request, and a 0.7 gate escalates to a reasoning model, sitting on the cost/quality frontier.
Jev CookbooksDocs#official#cookbook#extraction
Official
Catch wrong and hallucinated citations against RFC 7519 with a string match plus one Choice question: four accurate citations verified at 0.93 or more and all four planted failures caught.
Jev CookbooksDocs#official#cookbook#citations
Official
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.