Docs
CommunityJev Benchmarks & Evaluations
JEVWeb
xiq's JEV-format view of OpenJev-Data: every browser step re-expressed as state, questions and gold across eight configs covering Mind2Web, NNetNav, WebChain, two teacher runs and on-policy GLM, plus a frozen 2,346-step bench and 153 live tasks. Licences are mixed and per-row terms apply.
- Format
- Docs
- Reading time
- 2 min
- Published by
- Community
- Author
- xiq
- Last verified
- Oct 2026
- Languages
- Python
Highlights
- Eight configs over train/validation/test: mind2web 5,861/586/875, nnetnav_audited 11,408/1,140/1,150, webchain 17,876/1,069/1,589.
- Teacher and on-policy configs: teacher_qwen3max, teacher_glm and onpolicy_glm from DAgger-style runs.
- bench is a frozen 2,346-step OpenJev-Bench test set; live_tasks ships 153 held-out end-to-end tasks.
- Every row is state, questions and gold; JSONL mirrors each parquet, and parquet keeps objects as JSON strings.
- A step is correct when the operation and its target both match a candidate element from the state.
Quickstart
python
from datasets import load_dataset
m2w = load_dataset("xiq/jevweb", "mind2web")
nnetnav = load_dataset("xiq/jevweb", "nnetnav_audited")
bench = load_dataset("xiq/jevweb", "bench", split="test")Watch out
Card reviewed 2026-10-17; licences are mixed (Mind2Web and WebChain rows CC BY 4.0, NNetNav rows Apache-2.0, teacher and on-policy rows subject to website and provider terms), so commercial use needs a terms check. live_tasks rows carry no page state, and the OpenJev-Data card remains the authority on provenance, redaction and label auditing.
Questions about this entry
Short answers drawn from the linked project or page.
JEV-format views of alanhuangya/OpenJev-Data: the same browser steps, one folder per config, with every step unwrapped into the three parts a decision model consumes - state, questions and gold - instead of the source's single request JSON string plus flat metadata columns.
state is a JSON string with page url, title, visible text, 8-40 candidate elements and recent actions; questions holds the typed questions, including an operation plus a target question per action that needs one; gold is the correct option key for each labelled question.
mind2web, nnetnav_audited with audited DONE labels, webchain v2, teacher_qwen3max, teacher_glm, onpolicy_glm, a 2,346-step bench and live_tasks with 1,402 train goals plus 153 held-out tasks. Each step config splits into train, validation and test.
A step is answered correctly when the operation and, if the operation takes one, its target both match gold. Targets are chosen among the candidate elements listed in the state, not among every element on the page, so scoring respects the options the model was offered.
The card says a typed struct would inject null keys into elements that the source omits (those without value or options), so parquet keeps the objects as JSON strings; the JSONL files keep the native nesting instead.
Mixed: Mind2Web and WebChain rows are CC BY 4.0, NNetNav rows Apache-2.0, while teacher, on-policy and live-task rows contain text captured from public websites and stay subject to those sites' and the labelling models' provider terms. Treat it as research data and check terms before commercial use.
Those rows hold only id, url, goal and source - no page state - so they cannot be expressed as state, questions and gold until an agent visits the site and the page is parsed into elements. The 153 test tasks are the held-out end-to-end suite.
More Jev resources like this
johannhartmann's German translation of the LocalLLaMA/typed-decisions benchmark at revision c806f6c: states, instructions, options and rubrics translated to German while keys, ids and gold distributions stay as in the original. Machine-translated without human review. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
326,619 typed decision records from the frozen community-hard-mix-v2-final mixture: a redistributable projection of the data behind Open-Jev 27B training, with 2,053 Wikispeedia rows omitted.
Jev Benchmarks & EvaluationsDocs#community#benchmark#evaluation
Community
The research record of the Xyntetik typed-decisions track: TD-000 measurements and the preregistered TD-001 and TD-002 tests of whether /v1/decide probabilities from Xyntetik-Kvist-14B are usable as decisions. Both tests are INCONCLUSIVE; the record holds questions, scored rows, analyses and the harness. Apache-2.0.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
Twelve frozen typed-decision configs from the independent Open-Jev project, publishing controlled and mostly synthetic tasks with probability or ordinal targets: from the 113,568-row default release to the 160,797-row browser and drone expansion. CC0 1.0; English, Chinese and Turkish.
Jev Benchmarks & EvaluationsDocs#community#evaluation#typed-decisions
Community
Twelve frozen, mostly synthetic typed-decision configs from the independent Open-Jev project, published as redistributable parquet splits with manifests, raw records and reconstruction instructions; the card states these are not official TypeSafe training data.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
m-lagnajit's typed-decision dataset v2.1: 21,591 short-text rows with a human gold label, a calibrated soft label and the raw teacher ingredients, plus v2.1's corrected label that swaps the teacher's wrong top picks.
Jev Benchmarks & EvaluationsDocs#community#evaluation#calibration
Community
From the Jev guides
Original write-ups on what this entry is for.
From the Jev AI model community
Posts from builders shipping with Jev right now.