A test run is a long series of small decisions: which case to try next, whether the result looks right, what a failure actually means. Those are the judgments that used to need a human or a brittle rule, and a wave of experiments now puts Jev in that seat. The early reports agree on both the promise and the catch.
The Playwright loop
The most detailed account is Jason Arbon's browser-testing loop. A Node controller holds a menu of probes, each mapped to a known Playwright operation, and offers only the ones that are executable in the current page state. Jev then makes two calls per probe: the first chooses which probe to run next, and the second judges whether the observed transition matches the requirement. Across 63 probes it made 126 Jev calls in 24.3 seconds of active testing, with a 131 ms median round trip, against a deliberately buggy storefront.
Two things are worth copying. The model never sees the screen — the controller sends page text and structured observations such as product names, quantities, and totals — and everything outside the two decisions is ordinary software. The available moves are defined in code, and so is the meaning of correct.
Mobile QA and CI
The same split shows up elsewhere. Callstack's mobile QA experiment paired Jev with its agent-device tool and reported a single run in 14 seconds for about $0.0023 in inference. On the CI side, jev-triage labels issues and pull requests with a confidence-gated action, and jev-harness turns a failed test step into a categorised annotation, with a deterministic offline engine that can skip the model call entirely when the answer is obvious.
Rules first, the model second
The clearest guidance comes from Katalon's QA write-up, which is blunt: use deterministic assertions wherever the expected result is precise, and reserve Jev for the gaps between them. Arbon reaches the same conclusion from the other direction, recommending assertions first and Jev only where a rule cannot decide.
That ordering matters because the failure modes are real. Arbon reports a normal checkout that Jev scored 0.49 for expected behavior against 0.47 for the bug — a near coin flip — and an earlier run that falsely flagged an ordinary add-to-bag. The typed schema stops a malformed answer, not a wrong one. A constrained choice can still be the wrong choice.
The operational caveats
Three more habits come out of the reports. Pin the model version, because jev-latest moves when a release ships and a regression suite should not have a silent variable. Decide what the pipeline does when the model is unavailable and default to unknown, never pass, so an outage cannot become a green run. And treat page text as untrusted input, since injected instructions in the content under test can reach the model.
A checklist
- Assertions first. Reach for Jev only where the expected result cannot be written as a rule.
- Pin the model version. Never run a regression gate against an alias.
- Default to unknown on failure. Never let an outage read as success.
- Measure against a labelled set. Compare Jev's verdicts with past failures your team already agreed on before trusting it in a loop.
- Set thresholds by risk. A gate that blocks a payment needs a higher bar than one that files a ticket.
A test loop is not where a decision model replaces testing; it is where it earns a narrow seat. The reports so far put Jev in the sequence-and-judgment role, with the assertions, the available moves, and the consequences firmly in code.
