甚麼是 AI 智能體測試？面向手機 App 團隊的實用定義

Disclosure: I work on Phonebox, which gives AI agents cloud Android phones. This post is about the testing approach, not the product; the last section shows one way to try it.

"Agentic testing" is on every testing vendor's homepage this year, and the definitions don't quite agree. Here's a practical one, from a mobile point of view.

The definition

Agentic testing means an AI agent runs the test: you state a goal in plain language, and the agent decides each next action from what it observes on screen, acts, checks the result and adapts until the goal passes or fails.

That separates it from two things it often gets confused with:

Scripted testing: the steps are fixed in advance, in Espresso, UI Automator, Appium or Maestro.
AI-assisted testing: AI writes or repairs a script (Copilot writing Espresso, self-healing locators), and the script then runs as usual.

Tricentis, Momentic, Katalon and Autify all draw the line roughly here. Forrester calls the category "autonomous testing platforms" and doesn't use the word "agentic" at all.

Three ways to automate a UI test

	Scripted	AI-assisted	Agentic
Who decides the steps	The engineer, in code	AI writes the code; it runs like any script	The agent, at run time
When the UI changes	The test breaks	AI repairs the locator	The agent finds another way
Repeatability	Exact	Exact	Different paths to the same goal
Mobile examples	Espresso, UI Automator, Appium, Maestro	Copilot writing Espresso, self-healing locators	Claude Code on a cloud phone, Journeys in Android Studio

Where it falls short

Non-determinism: two runs can take different paths, so a failure can't always be replayed step for step.
Cost and speed: every step is a model call, so it's slower and dearer than a script.
Gestures: Google lists long-press, double tap, multi-finger gestures and rotation among the gaps in its own Journeys preview.
Judgment: the agent only knows the business rules you write down.

So it isn't a replacement for the suite that gates your merges. It's good at the things scripts are bad at: new screens, exploratory passes and reproducing bug reports.

Trying it on an Android app

The agent needs a device it can read and act on. On a laptop that's an emulator plus a local MCP server like mobile-mcp. If the agent runs somewhere without an emulator, a hosted one works:

claude mcp add --transport http phonebox https://phonebox.dev/mcp \
  --header "Authorization: Bearer $PHONEBOX_API_KEY"

Then give it a goal, not steps:

Install the debug APK on my Phonebox phone and open it.
Sign in with the test account from TEST_ACCOUNTS.md, open each tab
in the bottom bar once, then sign out. Read the screen before every
action. Report any crash, error message or empty screen, quoting the
screen text. Park the phone when you finish.

The two habits that matter most: tell it to read the screen before every action, and ask for evidence (the exact screen text, or a recording) so the report can be checked.

Originally published on phonebox.dev. Curious how others are drawing the line between agent runs and scripted suites.