Cultivar

A CLI tool from the Pinecone DevRel team that helps you write tests for Agent Skills and docs, run them across coding agents, and iterate until they work. You can run the CLI directly or install the bundled skill and let your coding agent drive it.

Through an optional grading backend: cultivar grade latest --backend typesafe grades with TypeSafe System One instead of Claude. It needs the optional extra pip install 'cultivar[typesafe]' and a TYPESAFE_API_KEY in the environment, and the digest names System One rather than a specific Jev model.

The README describes the typesafe backend as about 30x cheaper and calls it good for CI gates, with the trade-offs stated: no written reasoning and no quoted evidence compared with Claude's grader. A --dry-run option previews what TypeSafe would be sent per section without spending anything.

With Python 3.11+ and uv: uv tool install cultivar, or install from source via the git URL. The packaged smoke test cultivar hello runs a tiny write-hello.py task; --no-grade exercises the runner without any API key, and it exits 0 on PASS with diagnostics on FAIL.

An Anthropic API key for the local grader, a Modal account for --remote sandboxes (recommended for parallelism and isolation), and per-agent credentials for local runs: Claude OAuth, a Copilot token with the Copilot Requests scope, or GEMINI_API_KEY for Gemini.

Results land in results/<timestamp>[__title]/ with the task definitions, per-runner structured results and stats, readable conversation traces, raw JSONL event streams, setup/verify/teardown logs, workdir files the agent wrote, and grades.json after grading. cultivar report and cultivar show summarize it.

Each runner advertises with-skill and without-skill, plus with-docs when a task declares context_refs; doc_versions expands into one with-docs:<label> run per version, and self-navigate tests whether the agent can find the right doc. Two deltas answer whether the skill does anything and whether it beats dumping docs.