AI 測試自動化最大的問題並非編寫測試

A few years ago, creating automated tests was one of the biggest bottlenecks in software testing.

You needed someone who understood the application, knew how to write Selenium or Playwright scripts, and had enough patience to deal with selectors, timing issues, and browser differences.

AI is making the first part considerably easier.

You can describe a user journey in plain English and have an AI agent generate a test. Some tools can even execute it, analyze failures, and attempt to repair broken steps automatically.

That's impressive.

But I think we're about to discover that creating tests was never the most expensive part of test automation.

Maintaining them was.

And AI doesn't automatically solve that problem. In some situations, it introduces new problems that are harder to detect.

Here's an example.

Imagine an automated checkout test that clicks the "Place Order" button.

A developer changes the application, and the original selector stops working.

The testing platform uses AI to find a replacement element. It selects another button, executes the action, and marks the step as successful.

Great. One less broken test.

Except the replacement button was "Save for Later."

The automation worked. The test may even pass if its final assertions are weak.

But it no longer verifies what it was supposed to verify.

This is why I believe the next major challenge in test automation isn't test generation.

It's knowing whether your automated tests are still testing the right things.

1. Self-Healing Is Useful, but It Needs to Be Explainable

Self-healing has become a popular feature in AI-powered testing platforms.

The basic idea is reasonable.

When an element changes its ID, class, or location in the DOM, the testing tool attempts to identify the correct replacement instead of failing immediately.

That can eliminate a significant amount of routine maintenance.

But there's a distinction between repairing a locator and changing what a test does.

Consider three situations:

Scenario A: A CSS class changes.

The application still has the same login button, with the same label and functionality. Only its CSS class changed.

An automated repair is relatively low risk, provided the replacement identifies the same control.

Scenario B: Two buttons have identical labels.

The application now contains two "Submit" buttons in different sections.

The AI chooses one.

Which one? And how do we know it made the right choice?

Scenario C: The workflow changes.

The application now requires an additional confirmation step.

The AI decides to skip that step or modify the test to accommodate the new behavior.

At this point, we're no longer talking about locator repair.

We're talking about changing the test's intended behavior.

Those decisions shouldn't be treated as equivalent.

A useful benchmark methodology for evaluating AI testing platforms separates three important capabilities:

• Self-healing transparency: Can you inspect what changed and why?
• Editable step recovery: Can someone review and modify the repaired step?
• Reviewer traceability: Can you identify who approved a change and whether it can be reverted?

These are practical requirements, especially for teams maintaining large test suites.

If a platform changes a test automatically, I want to see the original step, the replacement, the evidence behind the decision, and the final outcome.

A green checkmark isn't enough.

And if the platform cannot reliably distinguish between a harmless locator change and a change in application behavior, a human should be involved.

2. Sometimes the Test Is Flaky Because the UI Is Doing Exactly What It Should

One of the most frustrating things about browser automation is that a perfectly functional application can still produce unreliable tests.

Virtualized tables are a good example.

Imagine an application displaying 10,000 customer records.

Rendering every record in the DOM would be unnecessarily expensive. Many modern interfaces solve this by rendering only the rows needed for the current viewport.

As the user scrolls, rows are added, removed, or recycled.

This works well for performance.

But it creates an interesting problem for automated tests.

Suppose your test expects all 10,000 rows to exist in the DOM.

It fails.

The application isn't broken. The test simply made an incorrect assumption about how the interface works.

The same thing can happen with infinite scrolling, lazy-loaded content, and scroll restoration.

I've found that a useful way to approach browser automation is to separate user-visible behavior from implementation details.

For a virtualized table, I wouldn't start by asserting an exact number of DOM elements.

I'd verify that:

1. The expected record can be reached.
2. The record displays the correct data.
3. Selecting the record performs the correct action.
4. Navigating away and returning preserves the expected state, if that's part of the product's behavior.

There's a detailed guide to testing virtualized tables, infinite scroll, and scroll restoration without flaky viewport assertions that explores these problems.

The principle is applicable far beyond virtualized tables.

Test the behavior the user depends on, not the incidental details of how the browser renders it.

Playwright's own locator documentation recommends locating elements using roles, labels, text, and other meaningful attributes.

That's generally a better starting point than brittle CSS paths or positional selectors.

But even good locators can't fix a test that makes the wrong assumptions about the application.

3. Your Browser Cloud Is Part of Your Testing Architecture

Another area that gets underestimated is infrastructure.

It's easy to get a few Playwright tests running locally.

Running hundreds or thousands of them reliably across browsers, operating systems, and CI environments is a different problem.

Now you have to think about:

• Parallel execution limits
• Browser and operating system versions
• Failed-session recordings
• Network access to staging environments
• CI configuration and credentials
• Reproducing failures locally
• Infrastructure costs

These don't sound like test automation problems.

But they directly affect how much time engineers spend maintaining the test suite.

Consider a test that passes locally and fails on a remote browser.

Without useful debugging artifacts, someone may need to rerun the test, capture logs, reproduce the environment, and investigate what happened.

That can become expensive when failures occur frequently.

Both BrowserStack and Sauce Labs support Playwright execution in cloud environments. Their official documentation describes their respective integrations and execution models.

The interesting question is which one creates less operational work for your team.

This BrowserStack vs Sauce Labs comparison for Playwright teams evaluates them through setup friction, debugging artifacts, CI integration, and maintenance overhead.

I think that's a more useful approach than comparing marketing feature lists.

If I were evaluating either service, I'd take an existing test suite and measure four things:

I'd also make sure the comparison uses equivalent browser versions, test data, concurrency, and application builds.

Otherwise, you're not comparing infrastructure fairly.

And I'd avoid concluding that either provider is universally better without running the workload that actually matters to your organization.

The cheapest browser cloud on paper can become expensive if every unexplained failure consumes an engineer's afternoon.

4. Mobile Testing Needs Something AI Can't Always Provide: Judgment

Mobile testing introduces another layer of complexity.

You're no longer dealing with just browsers.

You have device models, operating system versions, permissions, network conditions, system dialogs, orientation changes, and interactions between native components and embedded web content.

Tools such as Appium provide an automation foundation for mobile applications.

But executing an automated mobile test and determining whether the application is ready for release aren't the same thing.

Imagine a banking application where a confirmation screen appears after a transaction.

The automated test verifies that the expected text is present.

It passes.

But on a particular device, the confirmation button is partially hidden behind the system navigation area.

Or the application displays a technically correct message that's so poorly formatted that users misunderstand it.

An assertion checking for text won't necessarily detect either problem.

And even when AI identifies a possible visual defect, someone needs to decide whether it's a genuine issue, an acceptable variation, or a false positive.

This is why human review still has a place in mobile QA.

A recent evaluation of mobile testing services combining AI, human review, and traceable evidence discusses services such as Applause, Testlio, Global App Testing, and HeadSpin.

The useful distinction is between finding a potential problem and producing evidence that allows someone to act on it.

For an actionable mobile defect report, I'd want:

• Device model and operating system version
• Application build identifier
• Steps to reproduce
• Expected and actual behavior
• Screenshot or video
• Relevant application or device logs
• Review status and responsible owner

Depending on the issue, I'd also want the network conditions, permissions, and app state captured.

Without that context, a developer might spend more time reproducing the defect than fixing it.

AI can help identify suspicious behavior and organize evidence.

But for ambiguous failures, a person still needs to make a judgment.

That doesn't make the automation less valuable.

It makes the overall testing process more reliable.

5. The Framework With the Best Developer Experience Isn't Necessarily the Cheapest to Maintain

Let's talk about the inevitable Cypress vs Playwright discussion.

Both are capable testing frameworks.

Both have useful debugging features.

And both can become expensive to maintain if your test architecture is poorly designed.

Cypress provides an interactive runner with command logs and snapshots that let developers inspect what happened during execution.

Playwright provides tracing capabilities that can capture actions, DOM snapshots, network activity, and console output.

You can see these capabilities directly in the official Cypress documentation and Playwright Trace Viewer documentation.

Neither approach is inherently wrong.

The tradeoffs become more interesting when you're testing an AI-powered application.

Imagine a customer support dashboard with an embedded AI assistant.

The assistant generates responses asynchronously. Sometimes it calls multiple backend services. Sometimes its responses change because of updated prompts or retrieved information.

You need to test whether the interface behaves correctly without assuming that every generated sentence will be identical.

You might also need to test iframes, authentication flows, multiple browser contexts, or network failures.

At that point, debugging ergonomics and framework capabilities become important.

This Cypress vs Playwright comparison for AI-powered web applications examines those tradeoffs, including network control, iframe handling, execution parallelism, and debugging.

My preference would be to evaluate both using the same application scenarios rather than treating either framework as a default winner.

And I'd separate two questions that often get mixed together.

How quickly can a developer create the initial test?

And:

How much work will it take to keep that test useful over the next year?

Those can have very different answers.

A convenient test runner is valuable.

But if the tests require substantial maintenance every time the application changes, the initial convenience may not matter much.

How I'd Actually Evaluate an AI Testing Platform

I don't think a vendor demonstration is enough to evaluate a testing platform.

Most demonstrations use applications and test cases selected to make the product look good.

That doesn't tell you how the product will behave when your application changes unexpectedly.

I'd rather run a small, controlled proof of concept.

Here's a practical evaluation plan.

Step 1: Select five representative scenarios

Choose scenarios that expose different types of difficulty.

These are proposed experiments, not published benchmark results.

The goal is to see what happens when the tool encounters something less predictable than a straightforward login flow.

Step 2: Establish the ground truth

Before running the tests, document the expected behavior.

For the ambiguous button scenario, identify which button is correct.

For the virtualized table, define which records should appear and what should happen when they're selected.

For the mobile scenario, specify what constitutes a real defect.

This matters because you can't evaluate whether an AI system made the right decision unless you already know what the right decision was.

Step 3: Run the same scenarios repeatedly

A single successful execution doesn't prove reliability.

I'd repeat each scenario across the relevant browser or device configurations, using controlled application builds and test data.

I'd record:

• Correct passes
• Correctly detected failures
• False failures
• Incorrect passes
• Time spent investigating failures
• Manual interventions
• Changes made by self-healing

That fourth category, incorrect passes, is particularly important.

A flaky test is annoying because it creates additional work.

A test that passes while checking the wrong behavior is dangerous because it creates false confidence.

Step 4: Score the evidence, not just the result

For each failure or automated repair, I'd use a simple scoring system.

0: Insufficient evidence

The tool reports success or failure without enough context to determine what happened.

1: Reviewable

The tool provides enough artifacts to understand the outcome, such as steps, screenshots, logs, or trace data.

2: Reproducible and auditable

A reviewer can reconstruct the behavior, inspect automated changes, identify the environment, and approve or reject the result.

This isn't an industry-standard benchmark. It's a proposed rubric for comparing tools consistently.

I'd use it alongside execution accuracy and maintenance effort, rather than replacing those measurements.

A tool can provide excellent debugging evidence and still execute the wrong action.

Both dimensions matter.

The Cost Calculation Most Teams Skip

Here's the part I think deserves more attention.

When companies evaluate test automation, they often concentrate on licensing costs and test creation speed.

But those are only part of the equation.

A more useful model is:

Total cost of test automation = Test creation + Maintenance + Infrastructure + Failure investigation + Human review

You can make that model more detailed by adding training, integrations, and migration costs.

But even this simplified version captures an important distinction.

Suppose Tool A generates tests almost instantly but regularly requires engineers to investigate ambiguous failures.

Tool B takes longer to configure but produces more predictable executions and better debugging artifacts.

Which is cheaper?

You can't answer that question from the subscription price alone.

You need data from your actual application.

And that data should include the cost of investigating failures, not just the number of tests created.

This is especially relevant now that AI is making test generation so accessible.

If creating a test becomes nearly effortless, teams will naturally create more tests.

But unless those tests are reliable, understandable, and maintainable, we're simply increasing the number of things that can break.

What I'd Prioritize Over the Next 12 Months

If I were responsible for improving an existing automation program, I wouldn't start with a goal of doubling the number of tests.

I'd focus on three things.

First, I'd make sure every critical test has meaningful assertions. A successful click isn't proof that the intended business process worked.

Second, I'd improve failure evidence. When something breaks, the responsible engineer should have enough context to investigate it without immediately rerunning the test.

Third, I'd put controls around AI-generated modifications. Any system that automatically changes tests needs a way to explain those changes, preserve the original intent, and allow review when the decision is ambiguous.

Only after those foundations were in place would I aggressively expand automated coverage.

Because the goal isn't to have the largest possible test suite.

It's to have a test suite that consistently tells you something useful about your application.

Final Thoughts

AI is going to make automated testing more accessible.

That's a good thing.

It means smaller teams can automate workflows that previously required dedicated automation engineers.

But I don't think the industry should confuse easier test creation with better software quality.

The hard questions haven't disappeared.

Are the assertions meaningful?

Are the tests checking the correct behavior?

Can failures be reproduced?

Can automated repairs be explained?

And can the team maintain the tests without spending an unreasonable amount of time on them?

A testing platform that helps answer those questions is valuable.

One that simply makes everything look green is considerably less useful.

I'd rather have 200 automated tests I can trust than 2,000 tests that I don't fully understand.

And I suspect that distinction is going to become more important as AI-generated tests become the norm.

About this article and its sources

I approach this topic from the perspective of working on test automation software, where the operational side of testing matters as much as test creation. That perspective can introduce a preference for maintainability and traceability, so I've tried to make the evaluation criteria explicit.

This is a technical analysis and proposed evaluation framework, not a new hands-on benchmark of the vendors mentioned. The comparisons linked throughout the article provide additional perspectives; official Playwright, Cypress, Appium, BrowserStack, and Sauce Labs documentation should be used to confirm specific supported capabilities and configurations.

Last reviewed: November 3, 2026.

What's been more expensive for your team: writing automated tests or maintaining them? I'd be interested to hear how that has changed since you started using AI.