A new tool showed up in the QA conversation
Over the past few months, we've fielded a version of the same question from several clients: "we saw a demo of an AI agent that just... clicked around our app and found bugs, with no test scripts at all — should we be doing that instead of our Playwright suite?"
The short answer is no, not "instead of." The longer answer is worth writing down, because the category is real and the confusion about what it replaces is worth clearing up before teams either overinvest or dismiss it outright.
Autonomous exploratory testing agents are a genuine new category: AI agents that navigate a web or desktop application on their own — clicking buttons, filling forms, following links, triggering state transitions — without a pre-written script telling them what to click next. They build a working model of the application as they go and use that model to decide what to try next. No one authored a test case for "what happens if I submit this form with an empty required field while a modal is still animating in" — the agent found that path on its own.
That's a different mode of operation from anything scripted testing does, and it's worth being precise about what's actually new here.
What's structurally different from Playwright or Cypress scripts
A scripted E2E test, however well written, follows a fixed path: navigate to /checkout, fill field A, click button B, assert C. The script author decided that path in advance based on what they know about the application. The test is deterministic — same input, same steps, same assertion, every run. That's exactly the property you want for regression gating.
An exploratory agent instead does something closer to:
loop:
observe current page state (DOM / accessibility tree / screenshot)
infer available actions (clickable elements, form fields, navigation)
rank actions by "have I seen this before" and "does this look meaningful"
pick an action, execute it
record new state, note anomalies (errors, console warnings, broken layout, dead ends)
update internal map of the app
There's no fixed script — the "test" is generated at runtime from the agent's model of the app, and that model changes every run as the agent explores different branches. This is closer to how a human exploratory tester works during a bug bash than to a QA engineer writing Playwright specs. The agent isn't executing your test plan faster; it's building its own test plan on the fly, which means it can stumble into paths nobody wrote down — a double-submit race condition, an orphaned state after a browser-back, a form that accepts an emoji in a numeric field and then breaks a downstream calculation.
Where this genuinely helps
The strongest use case is coverage discovery, not coverage proof. Three concrete scenarios where it earns its keep:
- New or fast-changing UIs. A feature just shipped and the test-authoring backlog hasn't caught up yet. Instead of shipping with zero automated coverage until someone finds time to write scripts, an exploratory pass against staging can surface obvious breakage same-day.
- Unfamiliar codebases. When we inherit a client's legacy app with no meaningful test history, pointing an exploratory agent at it for a few hours produces a starting map of what's fragile, faster than a human clicking through manually.
- Edge cases nobody thought to write down. Scripted tests only check what the author anticipated. An agent that tries "what if I click submit twice quickly" or "what if I resize the viewport mid-flow" catches classes of bugs that never made it onto a written test plan, because no one thought to plan for them.
The common thread: these are all situations where the bottleneck is deciding what to test, not executing a known test faster.
Where it does not replace deterministic testing
This is the part that gets glossed over in demos, so we'll be blunt about it.
Compliance and audit evidence. If a regulator or auditor needs proof that "when a user does X, the system does exactly Y, verified on this date," you need a deterministic, versioned test with a fixed assertion — not an agent's exploration log, which by design takes a different path on every run.
Precise regression assertions tied to business logic. "Does the invoice total equal line items plus tax minus discount, to the cent" is not something an exploratory agent reliably checks unless it's told exactly what correctness means for that calculation. Exploratory agents are good at noticing that a page threw an error or rendered something visibly wrong; they're much weaker at confirming that a specific numeric output matches a specific expected value.
Deterministic reproducibility for CI gating. A CI gate needs to pass or fail on the same criteria every time, with a fast, predictable runtime. An agent whose path varies run to run doesn't give you a stable green/red signal, and it can be slow and non-deterministic in exactly the way a merge-blocking gate can't afford to be.
In short: exploratory agents are good at finding problems. They are not, today, a good system of record for proving correctness. Those are different jobs, and conflating them is where teams get burned.
A realistic integration pattern
The pattern we recommend keeps the two layers separate and lets each do what it's good at:
- Deterministic suite (Playwright, Vitest, etc.) stays the CI gate of record. It runs on every PR, blocks merges, and its assertions map directly to known business requirements. Nothing about adopting exploratory agents should touch this.
- Exploratory agent runs as a separate, non-blocking discovery job — nightly, or pre-release, against a staging environment that mirrors production closely enough to be meaningful.
- Findings get triaged, not trusted. Anything the agent flags becomes a ticket with a reproduction attempt attached, reviewed by a human before it's acted on. Confirmed, reproducible findings become candidate scripted regression tests, folded into the deterministic suite so the same bug can't regress silently later.
Conceptually, the pipeline wiring looks like this:
# .github/workflows/ci.yml — blocking gate, unchanged
on: [pull_request]
jobs:
test:
steps:
- run: npm run test:unit
- run: npx playwright test # deterministic, must pass to merge
# .github/workflows/nightly-discovery.yml — separate, non-blocking
on:
schedule:
- cron: "0 3 * * *"
jobs:
explore:
steps:
- run: deploy-to-staging
- run: run-exploratory-agent --target https://staging.example.com
- run: create-tickets-from-findings --require-human-reviewThe discovery job never gates a merge. It feeds the backlog, and human-reviewed, reproducible findings graduate into the deterministic suite over time. The exploratory agent effectively becomes a source of new test cases, not a replacement for the tests you already trust.
Risks worth calling out plainly
False sense of "full coverage." An agent's exploration report can look thorough — dozens of pages visited, hundreds of interactions logged — without ever exercising the specific business-critical path you actually care about. "The agent explored the app for two hours and found nothing" is not the same claim as "the checkout flow works correctly," and treating it as equivalent is how gaps go unnoticed.
Non-determinism makes bugs hard to reproduce. If the agent hit a bug on a path it generated dynamically, reproducing that exact path later — for a developer to debug it — requires good session/action logging from the agent itself. Without that, "the agent found a bug last night" turns into an unreproducible bug report, which is worse than no report at all.
Findings need human review before they're trusted. Agents can misclassify normal application behavior as a defect, especially around intentional loading states, animations, or non-standard-but-correct UI patterns. Piping agent findings directly into a ticket queue without a triage step will erode trust in the tool within a few weeks.
Our take
We'd advise piloting this the same way we advise piloting any new tool in an existing QA process: additively, not as a replacement. Pick one staging environment, run an exploratory agent as a nightly job for a few weeks, and measure two things — how many genuinely new, reproducible issues it surfaces that your existing suite and manual QA missed, and how much triage overhead it costs your team to review its findings. If the signal-to-noise ratio holds up, graduate the pattern into your release checklist. If it doesn't, you've lost a few weeks of a background job, not your CI gate.
The category is worth watching and, in the right spot, worth adopting. It's a tool for finding what you didn't know to look for — not a replacement for proving what you already know needs to be true.