跳到正文
DEV Community· Nat·· 3 天前AI 評分43

Aico Grounding:將自然語言介面目標轉換為螢幕位置

Aico Grounding: Translating a Natural-Language UI Target Into a Screen Location

AI 導讀

Aico Grounding 已開源,這個本機定位元件可將自然語言介面目標轉換成螢幕位置,供 Agent 執行操作。它會識別畫面上的互動元素、為元素分配參照,並返回預測目標位置;即使介面語言不同,也能識別元素。在 Aico 硬件本機執行可減少雲端路由造成的延遲,並避免把螢幕內容傳到外部伺服器。

正文

Cover image for Aico Grounding: Translating a Natural-Language UI Target Into a Screen Location

Nat

Understanding a screenshot is not the same as knowing where to click.

That gap is GUI grounding.

We've open-sourced Aico Grounding, a local grounding component built to translate a natural-language UI target into a location on the screen.

Screenshot + instruction → predicted UI target.

[attach annotated screenshot]

For a real-device agent, this is one of the critical steps between:

seeing the interface
→ understanding the task
→ taking the right action

The grounding layer runs locally. It identifies interactive elements across the screen, assigns each one a reference, and returns the predicted target location for the agent to act on.

The screenshot above shows this running on a real mobile interface — GitHub mobile with the UI in Chinese. The grounding model identifies elements correctly regardless of UI language.

Why local grounding matters

Cloud-routed grounding adds latency on every action. For a physical device agent that may take dozens of actions in a single task, that adds up fast.

Running grounding locally on the Aico hardware keeps the action loop fast and keeps screen content off any external server.

What's next

More benchmarks and demos coming — including instruction → detected target → actual click end-to-end.

GitHub: github.com/AidenAI-IO/aico-grounding

來源:DEV Community · dev.to