眼不見,仍在心中:ReMo 為 Omni-LLMs 壓縮視覺 token
Out of Sight, Still in Mind: Token Compression for Omni-LLMs
ReMo 是一種免訓練方法,透過跨模態對齊和精簡文字描述壓縮視覺 token,降低 Omni-LLMs 推理時的輸入 token 成本。在兩種規模的 Qwen2.5-Omni 上,它減少 54% 輸入 token,準確度仍與完整 token 模型相若,在五項音訊視覺基準的平均準確度分別達其 101.2% 和 101.4%。
Published on Nov 17
Authors:
,
,
Abstract
ReMo, a training-free method, reduces inference input token cost for omni-modal large language models by compressing redundant visual tokens through cross-modal alignment and replacing object-level visual tokens with compact text descriptions, achieving comparable accuracy on audio-visual tasks.
The goal of this paper is to reduce the input token cost of Omni-modal large language models (Omni-LLMs) at inference time. Omni-LLMs reason jointly over audio, video and text, but the cost of the three streams is highly unbalanced: visual tokens account for the vast majority of the input, and are highly redundant. In this paper, we propose ReMo, a training-free framework that compresses visual tokens by redistributing their information across modalities: a visual token is kept only if its information appears nowhere else. ReMo achieves this in two ways: (i) it aligns audio and video in a shared space via closed-form calibration, and removes visual tokens already explained by the audio or by other visual tokens; and (ii) it replaces object-level visual tokens, identified by a lightweight object detector, with compact text proxies, short descriptions of each object and its location, conveying the same content in far fewer tokens. On Qwen2.5-Omni at two model scales, ReMo removes 54% of the input tokens with comparable accuracy. It performs comparably to the full-token model, reaching 101.2% and 101.4% of its average accuracy over five audio-visual benchmarks.
View arXiv page View PDF GitHub 1 Add to collection
Get this paper in your agent:
hf papers read 2607.21179
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2607.21179 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2607.21179 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2607.21179 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co