Question 1

Does injecting a suggestion actually make the agent complete tasks better?

Accepted Answer

Nobody knows yet, including the author. The outcome benchmark harness is complete, with real SWE-bench tasks, frozen Docker environments and a paired analysis tested against hand-computed answers, but the results file does not exist because it was not fabricated: two of the four runtimes cannot complete a headless run here. The README is careful to distinguish not run from not proven, where not proven would mean the data was collected and the interval spans zero. Until there is an interval, the project claims a routing improvement and no outcome claim.

Question 2

Holdout recall@K of 0.600 against a 0.90 target is a big miss. What is actually broken?

Accepted Answer

Retrieval, not routing, and it is a structural mismatch rather than a bad parameter. recall@K depends only on the shortlist, which is built entirely by BM25 and per-kind quotas, so it can be swept with no API calls at all; growing the shortlist from 13 to 21 entries raised recall from 0.600 to 0.650 and saturated there. BM25 matches tokens, so a capability is retrieved only when a prompt happens to share vocabulary with its description, and prompts are phrased as tasks while descriptions are phrased as capability statements.

Question 3

An earlier version of this reported 0.804 recall. Which number should I believe?

Accepted Answer

The lower one, and the retraction is the most useful part of the README. The 0.804 development figure was inflated because MCP descriptions contained private service URLs and BM25 was matching a server's name out of the URL. Sanitizing the corpus removed that text and the number fell to its honest value, 0.824 on the development set and 0.600 on holdout. The author's conclusion is worth keeping in mind for any retrieval benchmark you read: a result that depends on a leaked URL is not a result.

Question 4

What does installing it do to my machine, and what leaves it?

Accepted Answer

Install writes a hook for Claude Code, Codex, Pi and OMP, and for Pi and OMP it also writes an extension that runs with your full user permissions, about forty lines that spawn the CLI, read one line of JSON back and inject it. The README tells you to read it at ~/.pi/agent/extensions/skillful/index.ts before trusting it. On the network side the prompt goes to api.typesafe.ai as part of the routing request, which is the only call the tool makes; the route cache in ~/.cache/skillful/routes.json never stores your prompt or your key, and there is no telemetry.

Question 5

How does a prompt get routed, step by step?

Accepted Answer

The hook does a cache lookup by prompt hash and catalog fingerprint, with a hit injecting in under 250 ms. On a miss a local BM25 scan with per-kind quotas builds a shortlist of about 15 at no token cost, then one TypeSafe request asks a choice question over that shortlist plus none and per-candidate noul questions to rank the runners-up. At most one primary and two runner-ups are injected.

Question 6

How do I inspect or test routing without spending?

Accepted Answer

npx @mrgoonie/skillful route --prompt '...' --explain shows the decision with the shortlist and scores, catalog --summary lists the capabilities found, and eval --recall-only sweeps quotas offline with no key and no cost. eval --replay FILE scores against recorded responses.

Question 7

Can I stop it from sending my prompt?

Accepted Answer

Yes. Set SKILLFUL_UPLOAD_PROMPT=false and it routes on the locally built shortlist without transmitting the prompt to the API, trading some retrieval quality for privacy. The route cache in ~/.cache/skillful/routes.json never stores your prompt or key, and there is no telemetry.

Jev Skillful

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev community