跳到正文
Hugging Face Blog·· 2025-01-24精選AI 評分62

Hugging Face 為 smolagents 加入視覺語言模型支援

We now support VLMs in smolagents!

AI 導讀

Hugging Face 為 smolagents 加入視覺支援,使視覺語言模型可原生用於智能體流程。圖片可以在 agent.run() 時一次傳入,也可透過 step callback 動態加入 ActionStep;文章並以 helium 示範視覺網頁瀏覽智能體。

推薦理由

文章整理了 smolagents 接收初始圖片與逐步加入截圖的兩種方式,並以網頁瀏覽智能體示範如何用 callback 更新執行記憶。

正文 · 繁體中文
偽善的人啊,先從自己眼中取出樑木,然後才能看清楚,取出你兄弟眼中的木屑。《馬太福音》7:3–5

簡介

我們已為 smolagents 加入視覺支援,原生支援在代理式流程中使用視覺語言模型。

目錄

概覽

在代理式 AI 領域,許多能力都被視覺這道門檻擋在外面。網頁瀏覽便是一個常見例子:網頁包含豐富的視覺內容,單靠擷取文字無法完整還原,例如物件的相對位置、透過顏色傳達的訊息、特定圖示等。在這種情況下,視覺能力可為代理帶來極大優勢。因此,我們已為 smolagents 加入這項能力!

這項功能的預告:能完全自主瀏覽網頁的代理式瀏覽器!

以下是運作方式示例:

我們如何讓 smolagents 看見事物

🤔 我們要如何把圖像傳給代理?可以透過兩種方式傳送圖像:

  1. 你可以在代理啟動時,直接提供圖像。文件 AI 等用途通常會採用這種方式。
  2. 有時則需要動態加入圖像。例如網頁瀏覽器剛執行某項操作,需要查看該操作對瀏覽視窗的影響。

1. 在代理啟動時傳入圖像一次

若要一次傳入圖像,我們已支援在 run 方法中向代理傳入圖像清單:agent.run("Describe these images:", images=[image_1, image_2]) 。

這些圖像輸入會與你想完成的任務提示詞一同儲存在 task_images 屬性中;此屬性屬於 TaskStep。

執行代理時,圖像會傳送給模型。這在根據包含視覺元素的大型 PDF 採取行動等情況下特別有用。

2. 在每個步驟傳入圖像 ⇒ 使用回呼

如何把圖像動態加入代理的記憶?

要了解這一點,我們首先要理解代理的運作方式。

smolagents 中所有代理都以單一 MultiStepAgent 類別為基礎,它是 ReAct 框架的一種抽象。簡單來説,此類別會循環執行以下步驟,並把現有變數和知識按以下方式記錄在代理日誌中:

  • 初始化:系統提示詞會儲存在 SystemPromptStep,用戶查詢則記錄至 TaskStep。
  • ReAct 循環(While):
    1. 使用 agent.write_inner_memory_from_logs() 把代理日誌寫成 LLM 可讀的對話訊息清單。
    2. 將這些訊息傳送至 Model 物件以取得完成結果,再解析結果取得動作(ToolCallingAgent 會輸出 JSON 區塊,CodeAgent 則會輸出程式碼片段)。
    3. 執行動作並把結果記錄到記憶中(ActionStep)。
    4. 每個步驟結束時,執行 agent.step_callbacks 中定義的所有回呼函數。 ⇒ 我們正是在這裏加入圖像支援:建立一個把圖像記錄到記憶中的回呼!

下圖詳細説明這個流程:

如你所見,對於動態擷取圖像的用途(例如網頁瀏覽代理),我們支援把圖像加入模型的 ActionStep,儲存在 step_log.observation_images 屬性中。

這可透過回呼完成,回呼會在每個步驟結束時執行。

以下示範如何建立這種回呼,並用它建構網頁瀏覽代理。👇👇

如何建立具備視覺能力的網頁瀏覽代理

我們會使用 helium。它以 selenium 為基礎提供瀏覽器自動化功能,讓代理更容易操作網頁。

pip install "smolagents[all]" helium selenium python-dotenv

代理可以直接使用 helium,因此不需要專用工具;例如可直接使用 click("top 10") 點擊頁面上名為「top 10」的按鈕。 我們仍需建立一些工具協助代理瀏覽網頁:一個工具返回上一頁,另一個工具關閉彈出視窗;由於 helium 難以定位關閉按鈕(這些按鈕沒有文字),因此需要這些工具。

from io import BytesIO
from time import sleep

import helium
from dotenv import load_dotenv
from PIL import Image
from selenium import webdriver
from selenium.common.exceptions import ElementNotInteractableException, TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

from smolagents import CodeAgent, LiteLLMModel, OpenAIServerModel, TransformersModel, tool
from smolagents.agents import ActionStep


load_dotenv()
import os

@tool
def search_item_ctrl_f(text: str, nth_result: int = 1) -> str:
    """
    Searches for text on the current page via Ctrl + F and jumps to the nth occurrence.
    Args:
        text: The text to search for
        nth_result: Which occurrence to jump to (default: 1)
    """
    elements = driver.find_elements(By.XPATH, f"//*[contains(text(), '{text}')]")
    if nth_result > len(elements):
        raise Exception(f"Match n°{nth_result} not found (only {len(elements)} matches found)")
    result = f"Found {len(elements)} matches for '{text}'."
    elem = elements[nth_result - 1]
    driver.execute_script("arguments[0].scrollIntoView(true);", elem)
    result += f"Focused on element {nth_result} of {len(elements)}"
    return result

@tool
def go_back() -> None:
    """Goes back to previous page."""
    driver.back()

@tool
def close_popups() -> str:
    """
    Closes any visible modal or pop-up on the page. Use this to dismiss pop-up windows! This does not work on cookie consent banners.
    """
    # Common selectors for modal close buttons and overlay elements
    modal_selectors = [
        "button[class*='close']",
        "[class*='modal']",
        "[class*='modal'] button",
        "[class*='CloseButton']",
        "[aria-label*='close']",
        ".modal-close",
        ".close-modal",
        ".modal .close",
        ".modal-backdrop",
        ".modal-overlay",
        "[class*='overlay']"
    ]

    wait = WebDriverWait(driver, timeout=0.5)

    for selector in modal_selectors:
        try:
            elements = wait.until(
                EC.presence_of_all_elements_located((By.CSS_SELECTOR, selector))
            )

            for element in elements:
                if element.is_displayed():
                    try:
                        # Try clicking with JavaScript as it's more reliable
                        driver.execute_script("arguments[0].click();", element)
                    except ElementNotInteractableException:
                        # If JavaScript click fails, try regular click
                        element.click()

        except TimeoutException:
            continue
        except Exception as e:
            print(f"Error handling selector {selector}: {str(e)}")
            continue
    return "Modals closed"

目前代理尚未接收視覺輸入。 因此,我們會示範如何透過回呼,把圖像動態送入步驟日誌。 我們會建立 save_screenshot 回呼,並在每個步驟結束時執行。

def save_screenshot(step_log: ActionStep, agent: CodeAgent) -> None:
    sleep(1.0)  # Let JavaScript animations happen before taking the screenshot
    driver = helium.get_driver()
    current_step = step_log.step_number
    if driver is not None:
        for step_logs in agent.logs:  # Remove previous screenshots from logs for lean processing
            if isinstance(step_log, ActionStep) and step_log.step_number <= current_step - 2:
                step_logs.observations_images = None
        png_bytes = driver.get_screenshot_as_png()
        image = Image.open(BytesIO(png_bytes))
        print(f"Captured a browser screenshot: {image.size} pixels")
        step_log.observations_images = [image.copy()]  # Create a copy to ensure it persists, important!

    # Update observations with current URL
    url_info = f"Current url: {driver.current_url}"
    step_log.observations = url_info if step_logs.observations is None else step_log.observations + "\n" + url_info
    return

最重要的一行,是把圖像加入觀察圖像清單:step_log.observations_images = [image.copy()]。

此回呼接受 step_log 和 agent 本身作為參數。把 agent 作為輸入,可執行比修改最近一筆日誌更深入的操作。

接着建立模型。我們已為所有模型加入圖像支援。 有一點要注意:使用 TransformersModel 搭配 VLM 時,初始化時必須把 flatten_messages_as_text 設為 False,才能正常運作,例如:

model = TransformersModel(model_id="HuggingFaceTB/SmolVLM-Instruct", device_map="auto", flatten_messages_as_text=False)

這個示範會透過 Fireworks API 使用較大型的 Qwen2VL:

model = OpenAIServerModel(
    api_key=os.getenv("FIREWORKS_API_KEY"),
    api_base="https://api.fireworks.ai/inference/v1",
    model_id="accounts/fireworks/models/qwen2-vl-72b-instruct",
)

接着定義代理。我們把 verbosity_level 設為最高,以顯示 LLM 的完整輸出訊息和思路;並把 max_steps 提高至 20,讓代理有更多步驟探索網頁。 我們亦會提供上文定義的 save_screenshot 回呼。

agent = CodeAgent(
    tools=[go_back, close_popups, search_item_ctrl_f],
    model=model,
    additional_authorized_imports=["helium"],
    step_callbacks = [save_screenshot],
    max_steps=20,
    verbosity_level=2
)

最後,為代理提供一些使用 helium 的指引。

helium_instructions = """
You can use helium to access websites. Don't bother about the helium driver, it's already managed.
First you need to import everything from helium, then you can do other actions!
Code:
```py
from helium import *
go_to('github.com/trending')
```<end_code>

You can directly click clickable elements by inputting the text that appears on them.
Code:
```py
click("Top products")
```<end_code>

If it's a link:
Code:
```py
click(Link("Top products"))
```<end_code>

If you try to interact with an element and it's not found, you'll get a LookupError.
In general stop your action after each button click to see what happens on your screenshot.
Never try to login in a page.

To scroll up or down, use scroll_down or scroll_up with as an argument the number of pixels to scroll from.
Code:
```py
scroll_down(num_pixels=1200) # This will scroll one viewport down
```<end_code>

When you have pop-ups with a cross icon to close, don't try to click the close icon by finding its element or targeting an 'X' element (this most often fails).
Just use your built-in tool `close_popups` to close them:
Code:
```py
close_popups()
```<end_code>

You can use .exists() to check for the existence of an element. For example:
Code:
```py
if Text('Accept cookies?').exists():
    click('I accept')
```<end_code>

Proceed in several steps rather than trying to solve the task in one shot.
And at the end, only when you have your answer, return your final answer.
Code:
```py
final_answer("YOUR_ANSWER_HERE")
```<end_code>

If pages seem stuck on loading, you might have to wait, for instance `import time` and run `time.sleep(5.0)`. But don't overuse this!
To list elements on page, DO NOT try code-based element searches like 'contributors = find_all(S("ol > li"))': just look at the latest screenshot you have and read it visually, or use your tool search_item_ctrl_f.
Of course, you can act on buttons like a user would do when navigating.
After each code blob you write, you will be automatically provided with an updated screenshot of the browser and the current browser url.
But beware that the screenshot will only be taken at the end of the whole action, it won't see intermediate states.
Don't kill the browser.
"""

執行代理

一切準備就緒:現在執行代理!

github_request = """
I'm trying to find how hard I have to work to get a repo in github.com/trending.
Can you navigate to the profile for the top author of the top trending repo, and give me their total number of commits over the last year?
"""

agent.run(github_request + helium_instructions)

不過要留意,這項任務相當困難:視乎你使用的 VLM,未必每次都能成功。Qwen2VL-72B 或 GPT-4o 等強大 VLM 的成功率較高。

後續步驟

這讓你初步瞭解具備視覺能力的 CodeAgent 能做甚麼,但還有很多工作要做!

期待看到你運用視覺語言模型和 smolagents 建構的作品!

來源:Hugging Face Blog · huggingface.co