Jev + WebMCP Breaks a Benchmark

Reported by the benchmark's author. The methodology and benchmark are open and reproducible, so you can rerun it yourself, but 49 tasks is a small corpus and the post is from the person who built it. The 112x cost figure is one configuration rather than a settled result.

Not well. Without WebMCP the authors' modified Browser Use Ultrafast setup solved 25 of 49 tasks; adding WebMCP took it to 49 of 49. They are explicit that 25 of 49 reflects their harness rather than a ceiling on Jev or on Browser Use, and they invite pull requests to improve it.

Because Jev cannot emit arbitrary text, which tool arguments require. It can pick search_products but cannot write the search query, so Mercury 2.5 generates arguments while Jev selects the tool. The authors' argument is that the real cognitive load is choosing the action, not filling in the form.

Because the win depends on sites exposing structured tools. The measured 18% cost reduction from adding WebMCP comes from a benchmark where every task site has a tool surface. On a site with no such tools you are back to picking buttons, at 25 of 49.

Because it compresses work. Without WebMCP the agent has to navigate menus, understand forms and recover from errors through a sequence of clicks; with it, one tool call replaces that sequence, so there are fewer decisions to pay for. The authors also cite TypeSafe's own note that Jev is weaker on multi-step reasoning, which WebMCP moves into the website.

They compare against two different Astra configurations: about 112 times lower model cost against Astra using computer use with code execution, and 245 times lower against Astra using screenshot-based computer use. The baseline matters, so the larger multiple is against the weaker of the two Astra setups.

Contributions to improve the harness. The authors say the 25 out of 49 result is specific to their modified Ultrafast setup and not a universal limit on Jev or Browser Use, and they link the benchmark repository for people to push the number higher. So the open methodology is also an open invitation to beat their own baseline.