Jev Directory

No. Evals are hand-authored with pinned expected verdicts, while community builds are user-generated and unverified, so the two collections carry different weight.

Quote the same manifest revision: it hashes every state, rubric, and expected verdict, and an eval passes only when every question matches exactly.

Point an MCP-compatible agent at the hosted endpoint for live search, or paste the capability pack and setup prompt into an agent for a static playbook.

Fork, edit the builds file or the paired eval and manifest files together, run the two build scripts, then open a pull request or use the short web form.

Each ships with a runnable prompt in boolean, choice or score question shape and a pinned expected verdict, so the manifest hashes every state, rubric and expected verdict and an eval passes only when every question matches exactly. That is what makes two people's numbers comparable: quote the same manifest revision. The questions come as an agent pack, an MCP endpoint exposing tools and resources, and an index file for static use.

Browse the directory of more than a thousand community builds, each entry carrying a link to the project and a link to the source post so you can check it independently before running anything. Builds are user-generated and unverified, unlike the hand-authored evals, so they carry different weight. To contribute, fork, edit the builds file or the paired eval and manifest files together, run the two build scripts that validate the manifest before emitting anything, then open a pull request or use the short web form.

Because it is what stops two people comparing results produced against different questions. The manifest hashes every state, rubric and expected verdict, and an eval passes only when every question matches exactly, so quoting the same manifest revision is the precondition for a fair comparison. If a rubric drifts, the revision changes and the old number no longer means what it did, which is why the directory ships the manifest alongside the runnable prompts rather than letting either be edited alone.