Question 1

Is this a fork of Jev Ultrafast?

Accepted Answer

No, and the README says so twice. It is a reproducible harness that drives an upstream checkout of browser-use/jev-ultrafast pinned at commit 452c1ad2dd628008f1d5608f28158d76e49e6cc0 and regenerates the report from fixtures/qc_rescored.json. The MIT line covers the harness only, and GitHub reports NOASSERTION for the repo.

Question 2

Can I see the results without a browser and without any API keys?

Accepted Answer

Yes, the reporting path is offline and stdlib-only on Python 3.10+. generate_report_v4.py rebuilds the field note from fixtures/qc_rescored.json and should produce a file over 20KB with the heading about where a research browser helps. The notebooks regenerate too: v1 over 40KB with a case id such as R2_doi_spelta_ot and the word Trace.

Question 3

Who decided whether a case passed or failed?

Accepted Answer

Human QC, recorded and locked. The baseline cases R1 to R11 and the stress suites S1 to S10 plus QS1 to QS8 and QS7b carry CoS-locked grades with history in the fixtures, and apply_qc.py merges auto results with those grades into qc_rescored.json. The grading rules themselves live in docs/METHODOLOGY.md rather than in the README.

Question 4

What does re-running the live suite actually require?

Accepted Answer

A checkout of jev-ultrafast at the pin with uv synced, Chrome with a CDP endpoint at BU_CDP_URL, plus TYPESAFE_API_KEY and TEXT_MODEL_API_KEY; pyyaml is optional for the case files. run_suite.py exits non-zero if any non-expect_fail case fails hard, but still writes every case JSON and a summary.json.

Question 5

How do I regenerate the report without any keys?

Accepted Answer

Run `python scripts/generate_report_v4.py --input fixtures/qc_rescored.json --output /tmp/jev_note_regen.html`. No browser or API keys are needed; expect a file over 20KB with the heading 'Where a research browser helps'.

Question 6

What upstream version does it pin?

Accepted Answer

It drives browser-use/jev-ultrafast at commit 452c1ad2dd628008f1d5608f28158d76e49e6cc0, headlined 'Reduce browser round trips and record a 7-second Flights demo'. It is a harness around that checkout, not a fork of Jev.

Question 7

What do the suites and notebooks cover?

Accepted Answer

Eleven baseline cases (R1 to R11), a human stress suite (S1 to S10) and a quant stress suite (QS1 to QS8 plus QS7b), all with CoS-locked QC grades and per-step Trace reports. Notebooks v1 and v2 regenerate offline from the fixtures.

Jev Research Eval

About this project

Highlights

Quickstart

Questions about this entry

Reactions & coverage

From the Jev guides

From the Jev AI model community