AI 智能體如何透過 ADB 操控實體手機：自動化實作中的難點
Giving an AI Agent a Real Phone: What ADB Automation Actually Looks Like
這篇實作記錄展示如何讓原本操作桌面的 AI 智能體透過 ADB 操控 Android 裝置，作者花了兩週，發現難點在追蹤介面狀態而非發送操作。ADB 可讀取 XML 檢視樹、注入操作事件並擷取螢幕，但 FLAG_SECURE 會令截圖為空，部分 Compose 和 Flutter 介面亦難以定位可點擊節點，非 ASCII 輸入則不適合直接使用 input text。
The browser was easy. The phone was humbling.
My agent already drives a desktop: screen, shell, browser. So pointing it at an Android device over ADB sounded like a weekend. It took two weeks, and every hard part was the same lesson as the desktop - the interface is the easy bit, state is the hard bit.
What ADB actually gives you
adb shell uiautomator dump returns the view hierarchy as XML with bounds for each node. adb shell input tap/swipe/text injects events. adb exec-out screencap -p gives you a PNG. That is enough to drive any app a human can drive by hand.
It is also enough to drive the wrong thing confidently.
Three walls you will hit
1. FLAG_SECURE screens return a zero-byte screenshot.
Password fields, banking apps, some onboarding flows. screencap exits 0 and writes nothing. If your loop treats an empty image as "screen unchanged", it will tap happily for ten minutes. Check the byte length and the PNG header, and treat a secure screen as a signal to switch strategy.
2. The UI tree lies about what is tappable.
Jetpack Compose and Flutter apps often expose a single node covering the whole screen, so every coordinate resolves to the same element. The fix is not bigger coordinates: use the keyboard (adb shell input keyevent) to move focus between fields, and verify by reading the tree back after every action rather than assuming the tap landed.
3. gettext does not exist in the shell.
Clipboard is the only sane way to type anything non-trivial: am broadcast -a clipper.set -e text "..." (or a clipboard helper), then input keyevent 279 to paste. Typing Chinese or any non-ASCII via input text is a trap.
The rule that saved the project
Never act on a cached view of the screen. Read, act, read again. It is slower per step and faster per task, because you stop discovering five steps later that the third tap went to the wrong window.
What it enables
With that in place the phone stops being a manual chore: the same agent that runs a shell script can open an app, log in once, and do the repetitive thing - not something you'd hand a stranger, but exactly the kind of work you resent doing yourself.
If you want the desktop side of this story: https://agent-home.ai
What is your go-to trick for driving an app that refuses to expose a usable accessibility tree?
來源：DEV Community · dev.to