如何用 Playwright（不使用 LLM）測試 AI 智能體能否找到定價頁

If you want to know whether an AI agent can get from your homepage to your prices, you can check most of it with a plain Playwright script and no model at all. The script looks for a link whose accessible name contains "Pricing", clicks it, waits for a price to render, and reads the prices from the page text. If any step fails, a browser agent driving your site through the accessibility tree or the DOM will probably struggle at the same spot.

I ran a first version of this on five browser-infrastructure sites today (Steel, Browser Use, Anchor Browser, Hyperbrowser and Browserless). All five passed. The script itself failed three times before it gave honest answers, and those failures are the useful part of this post.

What the check covers

Three questions, in order:

1. Is anything covering the page on load? The script counts visible elements with role="dialog" or aria-modal="true". A cookie wall or signup modal is the first thing an agent has to get past.
2. Is there a visible link named "Pricing"? It uses get_by_role("link", name=...), which matches the accessible name, the same thing a screen reader announces. An icon with no label, or a "Plans" item hidden inside a hover menu, won't match.
3. Are the prices plain text? It pulls every dollar amount out of innerText. Prices drawn in an image, a canvas, or hidden behind "Contact sales" don't show up.

It does not tell you how GPT, Claude or any specific agent behaves. It tells you whether the path is discoverable from the DOM, which is the floor every agent has to clear.

The script

import asyncio, json, re, sys
from playwright.async_api import async_playwright

PRICE = re.compile(r"\$\s?\d[\d,]*(?:\.\d+)?")

async def check(browser, url):
    ctx = await browser.new_context(viewport={"width": 1280, "height": 800}, locale="en-US")
    page = await ctx.new_page()
    r = {"site": url}
    try:
        await page.goto(url, wait_until="domcontentloaded", timeout=30000)
        try:  # busy sites with analytics may never go idle
            await page.wait_for_load_state("networkidle", timeout=10000)
        except Exception:
            pass
        # 1. Anything covering the page?
        r["open_dialogs"] = await page.locator("[role=dialog]:visible, [aria-modal=true]:visible").count()
        # 2. Is there a link an agent can find by its accessible name?
        links = page.get_by_role("link", name=re.compile("pricing", re.I))
        r["pricing_links"] = await links.count()
        visible = None
        for i in range(r["pricing_links"]):
            if await links.nth(i).is_visible():
                visible = links.nth(i)
                break
        r["visible_in_first_viewport"] = visible is not None
        if visible is None:
            r["result"] = "FAIL: no visible link named Pricing"
            return r
        await visible.click(timeout=8000)
        try:  # client-side routers change the URL after the click returns
            await page.wait_for_url(re.compile("pricing", re.I), timeout=10000)
        except Exception:
            pass
        try:  # wait for a price to render instead of sleeping a fixed time
            await page.get_by_text(PRICE).first.wait_for(timeout=10000)
        except Exception:
            pass
        r["landed_on"] = page.url
        # 3. Are the prices plain text, or locked in images / canvas / a sales form?
        text = await page.inner_text("body")
        r["prices_in_text"] = sorted(set(PRICE.findall(text)))[:12]
        r["contact_sales"] = bool(re.search(r"contact (sales|us)", text, re.I))
        r["result"] = "PASS" if r["prices_in_text"] else "WARN: no prices in page text"
        await page.screenshot(path=re.sub(r"\W+", "_", url) + ".png")
    except Exception as e:
        r["result"] = f"ERROR: {str(e)[:150]}"
    finally:
        await ctx.close()
    return r

async def main(urls):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        results = [await check(browser, u) for u in urls]
        await browser.close()
    print(json.dumps(results, indent=1))

asyncio.run(main(sys.argv[1:]))

Run it with pip install playwright && playwright install chromium, then python pricing_check.py https://yoursite.com/. Each site gets a JSON row and a screenshot of where the script ended up.

Three ways the script lied to me

1. It read the wrong page. My first version clicked "Pricing" and read the text right away. On steel.dev the result said it had landed on https://steel.dev/, the homepage, because the site's client-side router updated the URL after click() had already returned. Steel also lists prices on its homepage, so the check would have passed for the wrong reason. The fix is wait_for_url on a pattern before reading anything.

2. It timed out on a site that was fine. Waiting for networkidle with a 10-second limit threw a timeout on browserless.io and marked the whole site as an error. Some pages keep a request open the whole time, so "idle" never arrives. Treat networkidle as a best effort and move on.

3. It reported "no prices" on a page full of prices. After fixing the first two, Browserless came back as WARN: no prices in page text. An earlier version of the same check, which slept 3.5 seconds after each click, had found seven dollar amounts there. The pricing table renders after the document loads, and I was reading too early. Waiting for the first element whose text matches the price pattern fixed it, and Browserless passed.

My earlier version used fixed sleeps, so its timings measured my own waits more than the sites. That's why this post reports pass or fail and no load times.

The same trap applies to signup checks

We have an open bug in our own checker that's the signup version of mistake number 1. A rule that says "the agent found signup if it found an email input" counts a newsletter box in the footer as a successful signup. If you write a signup check like this, assert on something that only the real flow produces: the URL after submit, a "check your inbox" message, or a new row in your test database.

What a pass looks like

On today's run, all five sites had at least two links whose accessible name matched "Pricing", one of them visible without scrolling, no open dialog on load, and dollar amounts in the page text. Anchor Browser's link goes to a #pricing section on the homepage instead of a separate page, which still passes because the prices are readable text.

FAQ

Does a pass mean ChatGPT's agent will find my prices? No. It means the path is there in the DOM with names a machine can match. Real agents also deal with bot checks, slow pages and their own mistakes.

Why match on "pricing" only? Because it's the word most sites use. If your nav says "Plans", change the regex to pricing|plans and make sure the label is real text or an aria-label, not only an icon.

Should I run this in CI? It's cheap enough to run on every deploy against staging. A redesign that moves pricing into a hover menu would turn it red.

I'm building BotFumble, which runs checks like this on signup and checkout flows and records where an agent gets stuck. It isn't open yet. If you want to see the next runs, including the ones that fail, follow @VerigateAI on X, where I post each one with the script output.