A browser agent is a loop with a mouse. Every step is a decision (what to click, what to type, whether the page is ready, whether the goal is done), and the environment is adversarial. A page can change between observation and action, a label can lie, and text on the page can address the agent directly. Gating and recovery are not polish; they are the loop.
The action space is the safety boundary
The safest browser agents do not let the model invent actions: code builds a legal action list, and the model chooses from it. Jev Browser Skill does exactly that. One request per step carries goal_done, blocker, action, click_target, type_target, submit_after_type, and progress. The model selects among candidates code already filtered; targets are element IDs, and typed values come from your inputs.
The rest of the contract is code-owned. A --secret is typed into the page but never sent to the model. Success requires 0.85 confidence, or 0.7 on final verification, and the final page is verified before success is reported. A blocker at 0.6 hands the browser back for a login, CAPTCHA, or consent dialog. Step, cost, and time budgets are enforced outside the model, and code remembers tried and blocked edges.
The demos are the author's recordings: a three-step run cost $0.0003, a six-step Wikipedia run $0.0029 and 21 seconds, and a ten-step sign-in run $0.0013 and 18 seconds. At roughly a third of a cent per step, gating every step is affordable.
One request per step, with speculative targets
A step needs two dependent answers: which operation, and which element. Asking twice doubles round trips. Zurfyx's browser skill asks once: the request includes the operation plus a target for every operation, and code reads only the matching answer. After Jev returns TYPE, the click target is discarded and only the type target is read.
A question caps at 255 options, so the skill offers the first 250 controls and logs when the page is bigger. Jev sees a numbered table of visible controls from an in-page snapshot, never the DOM, and any answer outside the offered set throws. Measured on a MacBook, a Selenium form took five steps and 5.1 seconds, 1.4 seconds inside Jev; a Hacker News run took eight steps and 8.1 seconds, 2.2 seconds in Jev. A blank page costs about 1k input tokens and the Hacker News front page about 18k.
The debugging story is the best evidence. While building the login demo, Jev returned BLOCKED at 0.31 after the password step: the fields were named acct and pw, and both the link and button were called login, so the table gave it nothing to separate them. Better labels plus roles in the action history took the same situation to CLICK login at 0.89. Jev is stateless; the history code sends is its only memory.
It is a state-design lesson, not a model failure. The action space, the labels, and the history are the interface.
Grounding with boxes and images
When the DOM does not give you a target, an image question can. Box questions return one bounding box per question as four edges (top, bottom, left, right), each a fraction of the image, plus confidence. The same answer works at any image size. Box takes no criteria; instructions describe what to find. It needs jev-1.14.0 or jev-preview and the images header; jev-1.13.0 returns HTTP 400.
There is no not-found value: an absent target still returns four edges, so pair Box with a Noul presence check. No confidence threshold is recommended yet; tune one on your images and re-ask at higher resolution when confidence is low.
Images and assets are the other half. PNG or JPEG stills ride in a top-level assets map keyed by a reference string that can appear in state, instructions, or criteria. Limits are 256–1568 pixels per side, 16 MiB per file, and 32 MiB per request body. Both models accept images, but the page is experimental: it warns that all models perform worse on images than text and recommends the simplest possible questions. Images bill as input tokens.
Use vision only when the text representation is insufficient; a numbered control table is cheaper and more reliable.
Local models and recovery
Laya Browser Agent runs the open Laya model locally through MLX on Apple Silicon or PyTorch, and speaks a TypeSafe-compatible /v1/systemone dialect. Output is constrained to an operation plus an element index, validated against the offered set before acting; out-of-set keys, non-summing vectors, or non-argmax choices are rejected to fail open. A Playwright-backed loop adds element tables, loop guards, and model-free validation tests.
The documented failures are the useful part: auth fields, long or dynamic forms, collapsed menus, canvas and shadow-DOM content, headless-blocking sites, and language mismatches. Performance numbers are historical diagnostics with no guarantees, and offline accuracy is not end-to-end success. A local model buys availability, not immunity.
Recovery is code-owned: remember tried and blocked edges, detect loops, enforce budgets, verify the final page, and hand back on a blocker. The model can propose another action; the loop must know when to stop.
Page text is untrusted
HiddenLayer's security research is the cautionary case. On a simulated checkout page, an attacker-controlled advertisement caused the browser agent to click away from the purchase despite instructions to treat page text as untrusted. Targeted TokenBreak-style character edits lowered toxicity scores for prompts both Jev and the test model had initially flagged. The browser attack also worked against the conventional, text-generating version of the in-house model. The research is vendor-authored, Jev is black-box, and its architecture is undisclosed.
Decision models are not uniquely vulnerable; an instruction to ignore page text is guidance, not enforcement. Origin allow-lists, session isolation, confirmation for sensitive steps, and a final check are enforcement.
Security tooling is not action gating
Laya Browser Guard is a local-first Chrome extension that inventories a page's browser-visible attack surface and reports findings with severity, confidence, and evidence. It runs deterministic checks for mixed content, insecure forms, and missing headers, and can add five typed questions through an optional gateway. It never attacks a target or chooses browser actions; it is a different product from an agent gate.
Use the per-action discipline of confidence-gated routing: check_balance can run at 0.6, approve_transfer needs above 0.85, and high confidence is never permission. The jaggedness list adds local failure modes (counting, dates, interpolation, probabilities that do not sum), which is why budget and loop checks belong in code.
| Action class | Gate | Recovery |
|---|---|---|
| Navigate, scroll, observe | Allow | Retry once, then hand back |
| Fill a non-secret field | Allow with validation | Re-observe and re-ask |
| Click a destructive or payment control | Ask or block | Human confirmation |
| Enter credentials | Secret injection, never model output | Hand back at CAPTCHA or consent |
| Finish the task | Verify final page at the higher threshold | Continue if verification fails |
Gating and recovery in code
- Build the action list in code; let the model choose only from it.
- Validate output against the offered set before acting.
- Put secrets in the page, never in the request.
- Enforce step, cost, and time budgets outside the model.
- Verify the final state before reporting success.
- Treat page text as hostile and restrict origins and sessions.
For the broader survey, see Browser and Computer Use; for the injection model, Guardrails and Prompt Injection; and for image support, Jev 1.14, Images, Box, and the Danger Zone. The agent loop entry covers control flow, danger zone covers experimental features, and fail-open is a posture to choose deliberately.
