跳到正文
Hugging Face Blog·· 2025-07-10精選AI 評分60

Hugging Face 推出 ScreenEnv,用於測試和部署 GUI 智能體

ScreenEnv: Deploy your full stack Desktop Agent

AI 導讀

Hugging Face 推出 ScreenEnv,這是一個可在 Docker 容器中建立隔離 Ubuntu 桌面環境、用於測試和部署 GUI 智能體的 Python 程式庫。

推薦理由

ScreenEnv 將桌面控制、Docker 隔離與 MCP/API 接入整合在同一工具,並以 smolagents 範例展示如何建立和執行桌面任務智能體。

正文 · 繁體中文

摘要:ScreenEnv 是功能強大的 Python 程式庫,讓你在 Docker 容器中建立隔離的 Ubuntu 桌面環境,用於測試和部署 GUI 代理程式(亦稱電腦操作代理程式)。它內置支援 Model Context Protocol(MCP),讓部署可觀看、點擊並與真實應用程式互動的桌面代理程式變得前所未有地簡單。

甚麼是 ScreenEnv?



假設你需要自動執行桌面任務、測試 GUI 應用程式,或建構可與軟件互動的 AI 代理程式。以往這需要複雜的虛擬機設定和脆弱的自動化框架。

ScreenEnv 透過在 Docker 容器中運行的沙盒桌面環境解決這個問題。你可把它視為完整的虛擬桌面工作階段,程式碼能全面控制整個桌面體驗——不只點擊按鈕和輸入文字,還可管理桌面,包括啟動應用程式、整理視窗、處理檔案、執行終端機指令及錄製整個工作階段。

為甚麼選擇 ScreenEnv?

  • 🖥️ 全面控制桌面:完整的滑鼠和鍵盤自動化、視窗管理、應用程式啟動、檔案操作、終端機存取及螢幕錄影
  • 🤖 雙重整合模式:同時支援 AI 系統使用的 Model Context Protocol(MCP)及直接 Sandbox API,可配合任何代理程式或後端邏輯
  • 🐳 原生支援 Docker:毋須複雜的虛擬機設定,只需 Docker。環境彼此隔離、可重現,而且可在 10 秒內輕鬆部署至任何地方。支援 AMD64 和 ARM64 架構。

🎯 一行完成設定

from screenenv import Sandbox
sandbox = Sandbox()  # That's it!

兩種整合方式

ScreenEnv 提供兩種互補方式,可與代理程式及後端系統整合,讓你靈活選擇最適合架構的方式:

選項 1:直接使用 Sandbox API

最適合自訂代理程式框架、現有後端,或需要精細控制的情況:

from screenenv import Sandbox

# Direct programmatic control
sandbox = Sandbox(headless=False)
sandbox.launch("xfce4-terminal")
sandbox.write("echo 'Custom agent logic'")
screenshot = sandbox.screenshot()
image = Image.open(BytesIO(screenshot_bytes))
...
sandbox.close()
# If close() isn’t called, you might need to shut down the container yourself.

選項 2:整合 MCP 伺服器

最適合支援 Model Context Protocol 的 AI 系統:

from screenenv import MCPRemoteServer
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

# Start MCP server for AI integration
server = MCPRemoteServer(headless=False)
print(f"MCP Server URL: {server.server_url}")

# AI agents can now connect and control the desktop
async def mcp_session():
    async with streamablehttp_client(server.server_url) as streams:
        async with ClientSession(*streams) as session:
            await session.initialize()
            print(await session.list_tools())

            response = await session.call_tool("screenshot", {})
            image_bytes = base64.b64decode(response.content[0].data)
            image = Image.open(BytesIO(image_bytes))

server.close()
# If close() isn’t called, you might need to shut down the container yourself.

這種雙重方式讓 ScreenEnv 適應你現有的基礎設施,而毋須迫使你更改代理程式架構。

✨ 使用 screenenv 和 smolagents 建立桌面代理程式

screenenv 原生支援 smolagents,讓你輕鬆建立自訂桌面代理程式來執行自動化。只需幾個步驟,即可建立自己的 AI 桌面代理程式:

1. 選擇模型

選擇你希望用來驅動代理程式的後端 VLM。

import os

from smolagents import OpenAIServerModel
model = OpenAIServerModel(
    model_id="gpt-4.1",
    api_key=os.getenv("OPENAI_API_KEY"),
)

# Inference Endpoints
from smolagents import HfApiModel
model = HfApiModel(
    model_id="Qwen/Qwen2.5-VL-7B-Instruct",
    token=os.getenv("HF_TOKEN"),
    provider="nebius",
)

# Transformer models
from smolagents import TransformersModel
model = TransformersModel(
    model_id="Qwen/Qwen2.5-VL-7B-Instruct",
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)

# Other providers
from smolagents import LiteLLMModel
model = LiteLLMModel(model_id="anthropic/claude-sonnet-4-20250514")

# see smolagents to get the list of available model connectors

2. 定義自訂桌面代理程式

繼承 DesktopAgentBase,並實作 _setup_desktop_tools 方法,建立自己的操作空間!

from screenenv import DesktopAgentBase, Sandbox
from smolagents import Model, Tool, tool
from smolagents.monitoring import LogLevel
from typing import List

class CustomDesktopAgent(DesktopAgentBase):
    """Agent for desktop automation"""

    def __init__(
        self,
        model: Model,
        data_dir: str,
        desktop: Sandbox,
        tools: List[Tool] | None = None,
        max_steps: int = 200,
        verbosity_level: LogLevel = LogLevel.INFO,
        planning_interval: int | None = None,
        use_v1_prompt: bool = False,
        **kwargs,
    ):
        super().__init__(
            model=model,
            data_dir=data_dir,
            desktop=desktop,
            tools=tools,
            max_steps=max_steps,
            verbosity_level=verbosity_level,
            planning_interval=planning_interval,
            use_v1_prompt=use_v1_prompt,
            **kwargs,
        )

        # OPTIONAL: Add a custom prompt template - see src/screenenv/desktop_agent/desktop_agent_base.py for more details about the default prompt template
        # self.prompt_templates["system_prompt"] = CUSTOM_PROMPT_TEMPLATE.replace(
        #     "<<resolution_x>>", str(self.width)
        # ).replace("<<resolution_y>>", str(self.height))
        # Important: Adjust the prompt based on your action space to improve results.

    def _setup_desktop_tools(self) -> None:
        """Define your custom tools here."""
        
        
        @tool
        def click(x: int, y: int) -> str:
            """
            Clicks at the specified coordinates.
            Args:
                x: The x-coordinate of the click
                y: The y-coordinate of the click
            """
            self.desktop.left_click(x, y)
            # self.click_coordinates = (x, y) to add the click coordinate to the observation screenshot 
            return f"Clicked at ({x}, {y})"
        
        self.tools["click"] = click
        

        @tool
        def write(text: str) -> str:
            """
            Types the specified text at the current cursor position.
            Args:
                text: The text to type
            """
            self.desktop.write(text, delay_in_ms=10)
            return f"Typed text: '{text}'"

        self.tools["write"] = write

        @tool
        def press(key: str) -> str:
            """
            Presses a keyboard key or combination of keys
            Args:
                key: The key to press (e.g. "enter", "space", "backspace", etc.) or a multiple keys string to press, for example "ctrl+a" or "ctrl+shift+a".
            """
            self.desktop.press(key)
            return f"Pressed key: {key}"

        self.tools["press"] = press
        
        @tool
        def open(file_or_url: str) -> str:
            """
            Directly opens a browser with the specified url or opens a file with the default application.
            Args:
                file_or_url: The URL or file to open
            """

            self.desktop.open(file_or_url)
            # Give it time to load
            self.logger.log(f"Opening: {file_or_url}")
            return f"Opened: {file_or_url}"

        @tool
        def launch_app(app_name: str) -> str:
            """
            Launches the specified application.
            Args:
                app_name: The name of the application to launch
            """
            self.desktop.launch(app_name)
            return f"Launched application: {app_name}"

        self.tools["launch_app"] = launch_app

        ... # Continue implementing your own action space.

3. 讓代理程式執行桌面任務

from screenenv import Sandbox

# Define your sandbox environment
sandbox = Sandbox(headless=False, resolution=(1920, 1080))

# Create your agent
agent = CustomDesktopAgent(
    model=model,
    data_dir="data",
    desktop=sandbox,
)

# Run a task
task = "Open LibreOffice, write a report of approximately 300 words on the topic ‘AI Agent Workflow in 2025’, and save the document."

result = agent.run(task)
print(f"📄 Result: {result}")

sandbox.close()

如果遇到 Docker 存取遭拒錯誤,可以嘗試以 sudo -E python -m test.py 執行代理程式,或把你的使用者帳戶加入 docker 羣組。

💡 如需完整實作示例,請參閲 GitHub 上的 CustomDesktopAgent 原始碼。

立即開始使用

# Install ScreenEnv
pip install screenenv

# Try the examples
git clone git@github.com:huggingface/screenenv.git
cd screenenv
python -m examples.desktop_agent
# use 'sudo -E python -m examples.desktop_agent` if you're not in 'docker' group

下一步

ScreenEnv 計劃把支援範圍由 Linux 擴展至 Android、macOS 和 Windows,實現真正跨平台的 GUI 自動化。這將讓開發者和研究人員只需極少設定,便能建構可適應不同環境的代理程式。

這些進展為建立可重現的沙盒環境鋪路,非常適合用於基準測試及評估。

程式碼儲存庫:https://github.com/huggingface/screenenv

來源:Hugging Face Blog · huggingface.co