Question 1

Is the winning time real, or does the reference entry just plan before the clock starts?

Accepted Answer

Both halves are true and the README is explicit about them. The clock runs from Start run to Finish run and the page scores every event, but Almond-fastloop is allowed a planner pass before it, so part of its work is outside the timed window. Its token report shows the split: about $0.0018 of Jev calls against roughly $0.10 for one Claude planning call. The comparison is also across harnesses rather than like-for-like model calls: interactive Codex, Claude Cowork's inner browser and a cloud browser agent all ran the same five events.

Question 2

Can an entrant hard-code the answers and replay a winning run?

Accepted Answer

Each run draws its own identities at Start run — which athlete, which sport and list order, which 4-digit code, which safe-link wording — from a seed, while keeping the shape and amount of work identical so times stay comparable. A replayed script therefore fails the relay and slalom events. Ranking additionally requires a clean record chain: one start record, five events in order, no event under 300 ms, a total over 3 s, and client and server clocks within 4 s. Runs that fail are displayed as flagged rather than dropped.

Question 3

Is this a Jev benchmark, or a browser benchmark with one Jev entry?

Accepted Answer

It is a browser-agent benchmark with a Jev reference implementation. The events are generic: fill a three-field registration form, click the one link named Skeleton among sixteen sports, carry a 4-digit code between pages, reach the bottom of a very long page and confirm in a dialog, and avoid the big red button in favour of the small link. Jev appears because Almond-fastloop uses it for the per-tick choice; the other entrants are named tools and harnesses, and the Hall of Fame ranks runs rather than models.

Question 4

What does Jev actually decide per tick, and where does the loop refuse to act?

Accepted Answer

Each tick the loop reads the page through Chrome DevTools — interactive elements with names, values and a diff, about 2 ms — builds the list of executable options, asks Jev for one choice plus two yes/no questions covering whether the goal is complete and whether it is blocked, executes, and settles. Confidence below a threshold or a blocked verdict escalates to a planner once. The loop only types values it was given or numbers it has seen on earlier pages, and it stops at logins, captchas and payments.

Question 5

How do I run the reference loop myself?

Accepted Answer

Node 22 or newer. Open Chrome with --remote-debugging-port=9333 and a separate --user-data-dir, write TYPESAFE_API_KEY into ~/.config/typesafe/env with mode 600, then node olympics.mjs for the full course, which registers as Almond-fastloop, or node almond-fastloop.mjs task_booking.json for a single task. The optional planner shells out to the Claude Code CLI with claude -p; without it you pass the sub-tasks by hand.

Question 6

What were the first results, in seconds?

Accepted Answer

On the 2026-09-17 server clock, Almond-fastloop cleared all five events in 13.0, 13.4 and 13.8 seconds. Codex interactive with Sky computer use took 66.2 seconds, Claude Cowork's inner browser 100.3 seconds, and a cloud browser agent with an undeclared model 134.9 seconds. Cost is always reported in USD at public API list prices from bench/prices.md, and the 13.8-second run self-reported 78,534 input and 5,898 output tokens.

Question 7

What license is it, and how is the repository laid out?

Accepted Answer

MIT. almond-fastloop.mjs holds the loop and olympics.mjs the course runner, task_*.json are the individual tasks, olympics/ has build.py generating the site, hall.py publishing the Hall of Fame and the standard prompt.txt, and bench/ carries results, prices, the Codex rollout parser and raw traces. The benchmark itself is hosted and scored live on Almond, and a launchd job republishes the Hall only when results change.

Browser Use Olympics

About this project

Highlights

Questions about this entry

More Jev resources like this

From the Jev guides

From the Jev community