Question 1

Are the numbers real, or were the prompts picked after the fact?

Accepted Answer

The method is written down in EXPERIMENT.md, and the split discipline is explicit: formulations for the Jev and Haiku arms are chosen on the dev split before the test split is reranked, and candidates are frozen at 100 per query by the retrieval step. Pooled results are graded by Opus with a hand-graded agreement sample on top. Every script appends to its output, so a partial run is visible rather than silently replaced. The captured README documents the method; the results themselves are rendered at /report from a generated results file.

Question 2

What does reproducing the benchmark actually cost me?

Accepted Answer

The app needs only TYPESAFE_API_KEY. The comparison arms need a Claude Code subscription, because the Haiku arm and the Opus judge both run through your local claude CLI and are used by the benchmark only, never by the app. Haiku's pilot settled on ten prompts of ten cards per search, which puts the full test split at roughly 2,000 Haiku calls with eight in flight, plus a few hundred Opus calls for grading.

Question 3

Will this deploy to Vercel as written?

Accepted Answer

Not without the trace pinning, which is the most transferable part of the README. The search route reads three things invisible to the import graph, so next.config.ts pins them with outputFileTracingIncludes: the company data and index at 20.1 MB, the bge-small-en-v1.5 weights at 34.7 MB that nothing imports, and onnxruntime-node plus onnxruntime-common at 71.8 MB reached through a runtime require. That takes the traced function to 177.0 MB against the 250 MB limit.

Question 4

How do I know the bundle is right if local runs always pass?

Accepted Answer

A local run resolves everything from the full node_modules on disk, so it passes no matter what the trace left out, which is why the first deploy returned 500 on every search. The bundle check copies exactly the traced files into a sandbox and embeds a query there under node, and that is the only check that can catch a missing file. Run bun run build then bun run check:bundle rather than trusting a green dev server.

Question 5

How do I run the app locally?

Accepted Answer

bun install, copy .env.example to .env.local and set TYPESAFE_API_KEY, then bun run dev. The app needs only TYPESAFE_API_KEY; the Haiku arm and the Opus judge run through your local claude CLI and are used by the benchmark only, never by the app.

Question 6

What does the search route do?

Accepted Answer

Your query is routed by Jev, retrieved with BM25 plus embeddings, then reranked by Jev, over all 6,245 YC companies. The /report page renders the benchmark write-up from src/generated/results.json.

Question 7

What happens if the embedding model cannot load?

Accepted Answer

Search still answers: retrieve() drops the dense half, returns the BM25 ranking, and reports mode keyword_only, which the page prints under the results. It is a real loss, with grade-2 companies reaching the top 100 falling from 910/910 to 709/910.

Yc Jev Bench

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community