Organization repository on GitHub
A WebMCP benchmark, measures WebMCP against other browser-agent interfaces.
- Forks
- 7
- Last pushed
- Updated Sep 2026
Repo stats from the GitHub API, cached Oct 2026.
CommunityJev Practices & Patterns
WindTunnel
WindTunnel is a benchmark harness for browser agents. It runs the same tasks on the same self-hosted Docker sites across every interaction mode, WebMCP, code execution, DOM, accessibility tree and screenshots, and scores the resulting application state instead of judging the run.
- Stars
- 87
- Language
- HTML
- License
- APACHE-2.0
- Published by
- Community
- Author
- nekuda-ai
- Last verified
- Oct 2026
About this project
Benchmark WebMCP against other methods browser agents use to interact with websites. WebMCP solves 100% of tasks, with 2.5–7.5× faster median execution, 3–47× lower median cost, and 27–50% higher scores than the median of other methods. Quick start · Results · Run data · Methodology · Cost · WebMCP spec. WindTunnel compares WebMCP with other ways browser agents interact with websites. It runs the same tasks on the same sites and measures success rate, execution time, token usage, and cost.
Quoted from the project's README. Read the full README.
Used inJev for Software & DevTools
- Languages
- TypeScript
Highlights
- Outcome-based scoring: no human or model judge decides whether an attempt succeeded, the application state does.
- Each site is a real open-source application pinned to a fixed version in Docker, not a synthetic page.
- The canonical board covers 21 configurations across 49 tasks and 8 sites, with a 600s cap per attempt.
- A keyless CI mode, `WT_FAKE_LIFECYCLE=1 npm run bench`, runs the harness but scores 0/7 by design.
Watch out
The keyless mode needs Node 20.11 or newer and no Docker, but the real benchmark needs Docker on Linux or macOS plus a model API key. The exact model version served cannot be reported by the Stagehand and browser-use harnesses, so those rows are marked unavailable and the configured model is confirmed separately with scripts/verify-model.mjs. Coverage is eight specific test applications, not the open web, and the README does not document how to add your own sites or tasks. Copy reviewed against the repository README on 2026-10-04, not from running it.
Questions about this entry
Short answers drawn from the linked project or page.
The screen-driving arms are deliberately given roughly three times larger turn budgets, because the README argues a screenshot agent needs about three model turns per journey step. Even so, none of the 1,323 WebMCP attempts exhausted its budget, against 211 of 1,764 screen-driving attempts, and matched per model at 882 attempts a side the count is 0 against 124.
As a smoke test, yes. With Node 20.11 or newer and no key or Docker, `WT_FAKE_LIFECYCLE=1 npm run bench` exercises the harness without an LLM and scores 0/7 by design, so it only proves the thing runs. Anything real needs Docker on Linux or macOS, Playwright's Chromium, and your own model key, with `--arms` selecting which configurations run.
No, and the README concedes the point: ten configurations solve 49/49 tasks, including all nine WebMCP ones plus a code-execution run, so raw solve rate does not separate them. The argument rests on cost and time instead. For the same model, code-execution attempts cost $35.53 against $3.33 for native WebMCP, at 16.4s versus 6.3s median agent time.
The quick example runs two model arms on seven tasks across three lightweight sites once each, which is 14 attempts and well under a dollar, and `--budget` stops launching further attempts once accumulated spend passes the cap. The full canonical board is 21 configurations across 49 tasks with three attempts each, so size that budget separately rather than assuming the quick example scales.
It scores the resulting application state. No human or model judge decides whether an attempt succeeded, which is the outcome-based scoring the harness is built around.
Eight real open-source applications pinned to fixed versions in Docker, not synthetic pages, across 49 tasks with a 600-second cap per attempt.
Docker on Linux or macOS, Playwright's Chromium, and your own model API key. The keyless CI mode needs only Node 20.11 or newer and no Docker.
More Jev resources like this
11GitHub stars
Python browser agent that reads one lean page view per state and answers with a single JSON action, using any OpenAI-compatible model with no decision API or browser daemon.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
idovmamane/dejevuPython
idovmamane/dejevuPython7GitHub stars
Word-level language model experiment that uses Jev as its output layer, with explicit tokenizer, sampler, repetition mask, stop rules, and chunk verification.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
y0usaf/jev-lmTypeScript
y0usaf/jev-lmTypeScript7GitHub stars
Python client and decision toolkit for the Jev System One endpoint with noul, choice, and score primitives, batched asks, composition helpers, concurrent evaluation, and an MCP server.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
docxology/daf-jevPython
docxology/daf-jevPython2GitHub stars
An MCP server with two tools for agent memory: record writes facts into a per-project ledger as they happen, and recall scores each one against the task you are starting and hands back only what matters.
Jev Practices & Patterns#community#awesomejev#catalog-import
Community
Dharundp6/jev-carryforwardTypeScript
Dharundp6/jev-carryforwardTypeScript0GitHub stars
A small Node.js TypeScript CLI for trying Jev on synthetic support text: one urgency question, three questions in a single request, or a latency comparison. No web server and no UI.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
Menny1337/jev-labTypeScript
Menny1337/jev-labTypeScript1GitHub stars
Measures whether Jev can catch malicious agent skills on MalSkillBench, with a cascade arm that sends only the uncertain band to an LLM. Per-sample records, thresholds and calibration are all kept.
Jev Practices & Patterns#community#jevbest#catalog-import
Community
MohibShaikh/jev-skillbenchPython
MohibShaikh/jev-skillbenchPythonFrom the Jev guides
Original write-ups on what this entry is for.
Most Jev accuracy problems are state problems. What belongs in a state, what is a pre-made judgment to leave out, and why naming the slice you ask about matters more than the wording.
Jev Practices & Patterns#state#design#primitives
Read article
A Noul returns one number and no separate confidence. How to read the value, pin its boundary with criteria, split compound questions, and set a per-action threshold that matches the cost of being wrong.
Jev Practices & Patterns#noul#primitives#confidence
Read article
A Score is a weighted position, not a ruler. Why the number can land between levels, why levels are judged separately, and how to combine several atomic scores in code instead of one crowded rubric.
Jev Practices & Patterns#score#primitives#scoring
Read article
From the Jev community
Posts from builders shipping with Jev right now.