座標點擊是陷阱：如何不靠猜測像素操作 GUI Agent

The first version of my GUI agent clicked by pixel coordinates. It worked in demos and failed constantly in practice, and the reason is boring: pixels do not move, but content does.

Here is what I changed, in order of impact.

1. Read the tree before you click

Both desktop and mobile expose an accessibility tree: uiautomator dump on Android, UI Automation on Windows, the accessibility APIs on macOS. Nodes come with labels, roles and bounds. Clicking the node whose text is "Send" survives a re-layout; clicking (984, 183) does not.

The tree is not a luxury. It is the difference between an agent that works on your machine and an agent that only works on the screenshot you tuned it against.

2. When the tree lies, use the keyboard

Some toolkits (I am looking at you, Compose and Flutter) expose one giant node for the entire screen. Every tap resolves to the same element. Coordinates do not save you here either.

What does work: move focus with TAB / arrow keys, confirm the focused field by reading the tree again, then insert text. Typing into "whatever has focus" is fragile; typing into "the field the tree says is focused" is not.

3. Verify after every action, not after every task

Read → act → read. It feels wasteful. It is faster overall, because you stop discovering five steps later that step three landed on the wrong window and everything after it was theatre.

The check is cheap: did the screen change at all, did the expected text appear, did the node count change. One boolean per step.

4. Treat "nothing happened" as an error

An empty diff is ambiguous: either the action was a no-op, or it silently failed, or the UI has not repainted yet. Make the agent wait for a specific predicate ("the button is gone", "the row exists") with a timeout, and fail loudly when it never becomes true. Silent no-ops are how agents burn ten minutes of your day and produce a confident lie.

5. Never trust a cached screen

Screenshots go stale in under a second. Cache the plan, not the pixels.

None of this is model work. It is plumbing, and it is where reliability actually comes from. A mediocre model with tree-based selection, focus verification and predicate waits will beat a great model clicking blind - every time I have tested it.

The desktop and phone halves of this setup are described in the earlier posts in this series. Project page if you want the whole picture: https://agent-home.ai

Which failure mode has cost you the most debugging time: stale screens, ambiguous no-ops, or a tree that reports the whole screen as one node?