Hugging Face 推出 ScreenEnv,用於測試和部署 GUI 智能體
ScreenEnv: Deploy your full stack Desktop Agent
Hugging Face 推出 ScreenEnv,這是一個可在 Docker 容器中建立隔離 Ubuntu 桌面環境、用於測試和部署 GUI 智能體的 Python 程式庫。
ScreenEnv 將桌面控制、Docker 隔離與 MCP/API 接入整合在同一工具,並以 smolagents 範例展示如何建立和執行桌面任務智能體。
摘要:ScreenEnv 是功能強大的 Python 程式庫,讓你在 Docker 容器中建立隔離的 Ubuntu 桌面環境,用於測試和部署 GUI 代理程式(亦稱電腦操作代理程式)。它內置支援 Model Context Protocol(MCP),讓部署可觀看、點擊並與真實應用程式互動的桌面代理程式變得前所未有地簡單。
甚麼是 ScreenEnv?
假設你需要自動執行桌面任務、測試 GUI 應用程式,或建構可與軟件互動的 AI 代理程式。以往這需要複雜的虛擬機設定和脆弱的自動化框架。
ScreenEnv 透過在 Docker 容器中運行的沙盒桌面環境解決這個問題。你可把它視為完整的虛擬桌面工作階段,程式碼能全面控制整個桌面體驗——不只點擊按鈕和輸入文字,還可管理桌面,包括啟動應用程式、整理視窗、處理檔案、執行終端機指令及錄製整個工作階段。
為甚麼選擇 ScreenEnv?
- 🖥️ 全面控制桌面:完整的滑鼠和鍵盤自動化、視窗管理、應用程式啟動、檔案操作、終端機存取及螢幕錄影
- 🤖 雙重整合模式:同時支援 AI 系統使用的 Model Context Protocol(MCP)及直接 Sandbox API,可配合任何代理程式或後端邏輯
- 🐳 原生支援 Docker:毋須複雜的虛擬機設定,只需 Docker。環境彼此隔離、可重現,而且可在 10 秒內輕鬆部署至任何地方。支援 AMD64 和 ARM64 架構。
🎯 一行完成設定
from screenenv import Sandbox
sandbox = Sandbox() # That's it!
兩種整合方式
ScreenEnv 提供兩種互補方式,可與代理程式及後端系統整合,讓你靈活選擇最適合架構的方式:
選項 1:直接使用 Sandbox API
最適合自訂代理程式框架、現有後端,或需要精細控制的情況:
from screenenv import Sandbox
# Direct programmatic control
sandbox = Sandbox(headless=False)
sandbox.launch("xfce4-terminal")
sandbox.write("echo 'Custom agent logic'")
screenshot = sandbox.screenshot()
image = Image.open(BytesIO(screenshot_bytes))
...
sandbox.close()
# If close() isn’t called, you might need to shut down the container yourself.
選項 2:整合 MCP 伺服器
最適合支援 Model Context Protocol 的 AI 系統:
from screenenv import MCPRemoteServer
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client
# Start MCP server for AI integration
server = MCPRemoteServer(headless=False)
print(f"MCP Server URL: {server.server_url}")
# AI agents can now connect and control the desktop
async def mcp_session():
async with streamablehttp_client(server.server_url) as streams:
async with ClientSession(*streams) as session:
await session.initialize()
print(await session.list_tools())
response = await session.call_tool("screenshot", {})
image_bytes = base64.b64decode(response.content[0].data)
image = Image.open(BytesIO(image_bytes))
server.close()
# If close() isn’t called, you might need to shut down the container yourself.
這種雙重方式讓 ScreenEnv 適應你現有的基礎設施,而毋須迫使你更改代理程式架構。
✨ 使用 screenenv 和 smolagents 建立桌面代理程式
screenenv 原生支援 smolagents,讓你輕鬆建立自訂桌面代理程式來執行自動化。只需幾個步驟,即可建立自己的 AI 桌面代理程式:
1. 選擇模型
選擇你希望用來驅動代理程式的後端 VLM。
import os
from smolagents import OpenAIServerModel
model = OpenAIServerModel(
model_id="gpt-4.1",
api_key=os.getenv("OPENAI_API_KEY"),
)
# Inference Endpoints
from smolagents import HfApiModel
model = HfApiModel(
model_id="Qwen/Qwen2.5-VL-7B-Instruct",
token=os.getenv("HF_TOKEN"),
provider="nebius",
)
# Transformer models
from smolagents import TransformersModel
model = TransformersModel(
model_id="Qwen/Qwen2.5-VL-7B-Instruct",
device_map="auto",
torch_dtype="auto",
trust_remote_code=True,
)
# Other providers
from smolagents import LiteLLMModel
model = LiteLLMModel(model_id="anthropic/claude-sonnet-4-20250514")
# see smolagents to get the list of available model connectors
2. 定義自訂桌面代理程式
繼承 DesktopAgentBase,並實作 _setup_desktop_tools 方法,建立自己的操作空間!
from screenenv import DesktopAgentBase, Sandbox
from smolagents import Model, Tool, tool
from smolagents.monitoring import LogLevel
from typing import List
class CustomDesktopAgent(DesktopAgentBase):
"""Agent for desktop automation"""
def __init__(
self,
model: Model,
data_dir: str,
desktop: Sandbox,
tools: List[Tool] | None = None,
max_steps: int = 200,
verbosity_level: LogLevel = LogLevel.INFO,
planning_interval: int | None = None,
use_v1_prompt: bool = False,
**kwargs,
):
super().__init__(
model=model,
data_dir=data_dir,
desktop=desktop,
tools=tools,
max_steps=max_steps,
verbosity_level=verbosity_level,
planning_interval=planning_interval,
use_v1_prompt=use_v1_prompt,
**kwargs,
)
# OPTIONAL: Add a custom prompt template - see src/screenenv/desktop_agent/desktop_agent_base.py for more details about the default prompt template
# self.prompt_templates["system_prompt"] = CUSTOM_PROMPT_TEMPLATE.replace(
# "<<resolution_x>>", str(self.width)
# ).replace("<<resolution_y>>", str(self.height))
# Important: Adjust the prompt based on your action space to improve results.
def _setup_desktop_tools(self) -> None:
"""Define your custom tools here."""
@tool
def click(x: int, y: int) -> str:
"""
Clicks at the specified coordinates.
Args:
x: The x-coordinate of the click
y: The y-coordinate of the click
"""
self.desktop.left_click(x, y)
# self.click_coordinates = (x, y) to add the click coordinate to the observation screenshot
return f"Clicked at ({x}, {y})"
self.tools["click"] = click
@tool
def write(text: str) -> str:
"""
Types the specified text at the current cursor position.
Args:
text: The text to type
"""
self.desktop.write(text, delay_in_ms=10)
return f"Typed text: '{text}'"
self.tools["write"] = write
@tool
def press(key: str) -> str:
"""
Presses a keyboard key or combination of keys
Args:
key: The key to press (e.g. "enter", "space", "backspace", etc.) or a multiple keys string to press, for example "ctrl+a" or "ctrl+shift+a".
"""
self.desktop.press(key)
return f"Pressed key: {key}"
self.tools["press"] = press
@tool
def open(file_or_url: str) -> str:
"""
Directly opens a browser with the specified url or opens a file with the default application.
Args:
file_or_url: The URL or file to open
"""
self.desktop.open(file_or_url)
# Give it time to load
self.logger.log(f"Opening: {file_or_url}")
return f"Opened: {file_or_url}"
@tool
def launch_app(app_name: str) -> str:
"""
Launches the specified application.
Args:
app_name: The name of the application to launch
"""
self.desktop.launch(app_name)
return f"Launched application: {app_name}"
self.tools["launch_app"] = launch_app
... # Continue implementing your own action space.
3. 讓代理程式執行桌面任務
from screenenv import Sandbox
# Define your sandbox environment
sandbox = Sandbox(headless=False, resolution=(1920, 1080))
# Create your agent
agent = CustomDesktopAgent(
model=model,
data_dir="data",
desktop=sandbox,
)
# Run a task
task = "Open LibreOffice, write a report of approximately 300 words on the topic ‘AI Agent Workflow in 2025’, and save the document."
result = agent.run(task)
print(f"📄 Result: {result}")
sandbox.close()
如果遇到 Docker 存取遭拒錯誤,可以嘗試以
sudo -E python -m test.py執行代理程式,或把你的使用者帳戶加入docker羣組。
💡 如需完整實作示例,請參閲 GitHub 上的 CustomDesktopAgent 原始碼。
立即開始使用
# Install ScreenEnv
pip install screenenv
# Try the examples
git clone git@github.com:huggingface/screenenv.git
cd screenenv
python -m examples.desktop_agent
# use 'sudo -E python -m examples.desktop_agent` if you're not in 'docker' group
下一步
ScreenEnv 計劃把支援範圍由 Linux 擴展至 Android、macOS 和 Windows,實現真正跨平台的 GUI 自動化。這將讓開發者和研究人員只需極少設定,便能建構可適應不同環境的代理程式。
這些進展為建立可重現的沙盒環境鋪路,非常適合用於基準測試及評估。
來源:Hugging Face Blog · huggingface.co