Playwright 單獨用於智能體測試是否足夠？Vedha 加入逐步驗證層

More teams now let an AI agent test their web app: "test the checkout", "check the signup flow". The agent drives a browser through Playwright, clicks through the flow, and reports back: all tests passed.

The problem is how the agent decides that. It looks at the page. If the page shows the success message it expected, the test passes. But a page can show success while the app is broken underneath: the API returned a 500, the server rejected the request, or it stored nothing at all. The test passes, and the bug ships.

I wanted to measure how often that happens, and whether a testing layer could catch these bugs without the agent (or a human tester) writing extra assertions. The result is Vedha, an open-source testing framework and MCP server built on Playwright.

A broken feature that passes its test

Here is an order form with a seeded bug: the order API returns 500 and stores nothing, but the page still shows "Order placed". The same test, with the same check (wait for "Order placed"), is run twice:

Playwright  PASS   "Order placed" is on the page
Vedha       FAILED declared checks passed, but the page reported: POST /api/orders returned 500
Server      0 orders stored

You can run it yourself in about 3 seconds: npx tsx examples/demo/run.ts in the repo.

A careful tester would catch this in Playwright too, by asserting on the network response. Most tests don't, and agents almost never do.

What Vedha adds to Playwright

Vedha keeps what Playwright does best (auto-waiting, locators, the browser itself) and adds a verification layer to every test step:

• Every step gets a result: verified, failed, outcome_unknown, unverified or blocked. Never a silent pass.
• Actions that change data are checked. Mark a step as mutating and Vedha watches the requests it sent. A 5xx, a rejected request or a page error fails the step even when the on-screen check passed. Where the API follows REST conventions, it also reads the data back to confirm it was stored.
• No duplicate side effects from retries. If a click times out mid-submit, Vedha won't click "Place order" again and create a second order. It reports outcome_unknown until a read-only check settles it.
• Agent explorations become regression tests. A verified flow can be exported as a plain @playwright/test spec for CI.
• Safe defaults for agents. The MCP server won't start without a domain allowlist, treats page text as untrusted data, and redacts secrets you pass to it.

The numbers, and their limits

Every number comes from raw trial files in the repo, and each held-out test set was written before any run on it.

AI agent testing (Playwright MCP vs Vedha MCP, same model, same held-out test tasks):

Scripted tests, no model:

• With typical text/URL-only checks, Vedha caught 60 of 140 seeded bugs; every Playwright and Selenium baseline caught 0.
• A 1,120-trial audit after the last change had 0 false passes and 0 false fails.
• Median overhead: 1.05x hand-written Playwright.

The limits:

• The AI-agent results come from one free model, at 3 trials per task. That's a pilot, not a verdict.
• When an expert writes every check by hand, Vedha and Playwright tie. Vedha's lead is with typical checks and with agents.
• Read-back needs a REST-style read endpoint: the same path, or its singular/plural sibling. GraphQL, separate read endpoints and filtered or per-user collections aren't covered.
• Vedha passed one fewer healthy flow than Playwright MCP on set 4, and it uses more tool calls.

Try it

With Claude Code:

claude mcp add vedha -- npx -y @vedhajs/mcp --allow-domain localhost

Then ask: "test the checkout flow on localhost:3000".

Any MCP client takes the same command (npx -y @vedhajs/mcp --allow-domain localhost). There is also an agent skill (npx skills add HMAKT99/VEDHA), a CLI for scripted tests (npm i -D @vedhajs/cli), and an SDK that wraps an existing Playwright page.

Repo, benchmarks and raw data: https://github.com/HMAKT99/VEDHA (Apache-2.0)

I'd most like to hear about real apps where it gets things wrong: a false failure on a healthy flow, or a bug it should have caught.