Aico Grounding:將自然語言介面目標轉換為螢幕位置
Aico Grounding: Translating a Natural-Language UI Target Into a Screen Location
Aico Grounding 已開源,這個本機定位元件可將自然語言介面目標轉換成螢幕位置,供 Agent 執行操作。它會識別畫面上的互動元素、為元素分配參照,並返回預測目標位置;即使介面語言不同,也能識別元素。在 Aico 硬件本機執行可減少雲端路由造成的延遲,並避免把螢幕內容傳到外部伺服器。
Understanding a screenshot is not the same as knowing where to click.
That gap is GUI grounding.
We've open-sourced Aico Grounding, a local grounding component built to translate a natural-language UI target into a location on the screen.
Screenshot + instruction → predicted UI target.
[attach annotated screenshot]
For a real-device agent, this is one of the critical steps between:
seeing the interface
→ understanding the task
→ taking the right action
The grounding layer runs locally. It identifies interactive elements across the screen, assigns each one a reference, and returns the predicted target location for the agent to act on.
The screenshot above shows this running on a real mobile interface — GitHub mobile with the UI in Chinese. The grounding model identifies elements correctly regardless of UI language.
Why local grounding matters
Cloud-routed grounding adds latency on every action. For a physical device agent that may take dozens of actions in a single task, that adds up fast.
Running grounding locally on the Aico hardware keeps the action loop fast and keeps screen content off any external server.
What's next
More benchmarks and demos coming — including instruction → detected target → actual click end-to-end.
來源:DEV Community · dev.to
