What does this benchmark measure?

How well different search methods pick the right tool for an LLM agent. It uses LiveMCPBench: 525 real MCP tools from 69 servers and 94 human-annotated tasks, where right tool first means the first result is a tool the task needs.

What were the headline results?

BM25 on the agent's query scored 32% right tool first at $0 per 1,000 searches; Voyage embeddings 46% at $0.01; FastMCP-style two-stage Jev search 56% at $1.03; Voyage rerank-2.5 over the whole catalog 60% at $2.23; embeddings top 100 then the experimental engine 59% with 92% in the top 5 at $0.26.

What is the experimental Jev engine?

experiments/engine.ts is a prototype of a search library whose ranking is a Jev decision, with Orama's shape. Against the AI SDK's mock evaluation model in tests, and on LiveMCPBench with embeddings prefiltering, it reached 59% right-tool-first with 92% in the top 5.

How do I reproduce a run?

Clone the repo, use Node 22+ and pnpm: pnpm install, copy .env.example to .env.local and add AI_GATEWAY_API_KEY or a VERCEL_OIDC_TOKEN, then run pnpm test with no network, or pnpm mcp --sizes 525 --limit 10 for a small resumable run. A full LiveMCPBench search-only run costs a few dollars at list price.

What is Jev's price in the benchmark?

Jev's input price is listed as $0.042 per million tokens, and Jev, the Voyage embeddings and the reranker all go through the Vercel AI Gateway. The two-stage Jev search arm cost $1.03 per 1,000 searches in the published table.

Why does the Claude end-to-end run use Vertex AI?

Because at the time it was written the gateway's Anthropic endpoint ignored the server-side tool-search tool, so those arms run on Google Vertex AI with ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION and gcloud auth instead.

Should I trust a four-point difference between two methods?

Carefully. The README says compare rows from different runs with care and that with 94 tasks differences under about 10 points are within the noise; for example the FastMCP-style search scored 56% in one run and 54% in a later interleaved one. Full tables and confidence intervals are in research/00-benchmark-results.md.

Jev directory

CommunityJev Benchmarks & Evaluations

Jev Tool Search

A benchmark comparing BM25, embeddings, rerankers and Jev for tool search over 525 real MCP tools and 94 annotated tasks, plus an experimental Jev search engine. Jev's two-stage search found the right tool first 56% of the time.

See highlights Open github.com

Stars: 2
Language: TypeScript
License: MIT
Reading time: 3 min
Published by: Community
Author: kachar
Last verified: Oct 2026

kachar/jev-tool-search

User repository on GitHub

View on GitHub

Tool search for LLM agents: BM25 vs embeddings vs rerankers vs Jev on 525 real MCP tools, plus an experimental Jev search engine

Forks: 0
Last pushed: Updated Sep 2026

Repo stats from the GitHub API, cached Oct 2026.

Tagscommunity catalog-import logicrw benchmark tool-search mcp

Works forJev for Ranking Jev for Tool calling Jev for Evaluation

Used inJev for Software & DevTools

Languages: TypeScript
Platforms: MCP Vercel AI Gateway
Capabilities: Open sourceRequires API key

Highlights

Headline results: BM25 32%, embeddings 46%, two-stage Jev 56%, rerank-2.5 60%, embeddings plus the experimental engine 59% with 92% in the top 5.
Cost per 1,000 searches: $0 BM25, $0.01 embeddings, $1.03 two-stage Jev, $2.23 reranker, $0.26 for the experimental engine path.
Measured 2026-09-25/26 from Frankfurt through the Vercel AI Gateway; with 94 tasks the README warns differences under about 10 points are noise.
The experimental engine turns ranking into a Jev decision and takes Orama's shape; tests run it against the AI SDK's mock evaluation model.
Backs two articles and ships resumable runners for the 525-, 1,000- and 2,000-tool runs, with results landing as JSONL in results/.

Quickstart

bash

pnpm install
cp .env.example .env.local        # add AI_GATEWAY_API_KEY (or a VERCEL_OIDC_TOKEN)
pnpm test
pnpm mcp --sizes 525 --limit 10

Watch out

Grounded in the truncated repo README and GitHub metadata read on 2026-10-15. Rows come from different runs — in a later interleaved run the FastMCP-style search scored 54% — and the README itself cautions that differences under about 10 points are within noise at 94 tasks.

Questions about this entry

Short answers drawn from the linked project or page.

Reactions & coverage

Posts, threads, and videos about this entry from around the web.

X: Jev in front of the HeyGen MCP

X: Playwright CLI + Jev vs Playwright MCP

X: A local MCP server that cuts computer use from 23s to 0.8s. Cartwright built Cortex, a local MCP server where…

X: A side panel that drives any site's WebMCP tools

From the Jev guides

Original write-ups on what this entry is for.

MCP tool routing: Jev in front of the tool call

Oct 16, 2026

Tool menus can outgrow the context that reads them. The catalog's Jev-based routers and gates narrow the list: benchmarked tool search, per-tool scoring, fail-closed approvals, and WebMCP in the browser.

Jev Agents & MCP#mcp#tool-routing#agents

Read article

The trust ledger: measuring confidence your domain has earned

Oct 16, 2026

ASSAY-001 calibrated Jev on CLINC150 but found it overconfident on Banking77 — one audit, two verdicts. How calibration research measures the model-and-domain pair, discounts stated confidence, and signs the evidence.

Jev Benchmarks & Evaluations#calibration#evaluation#open-models