跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 26 天前AI 評分48

LOWN:以證書引導視覺路由,提升擴散視覺語言模型效率

Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

AI 導讀

LOWN 在擴散視覺語言模型解碼時,按預測是否一致決定何時處理圖像,並以獲得證書的 token 數量決定每步解除遮罩的 token 數量。在十個基準測試中,無需訓練或調校閾值,LOwn 平均只用 35.3% 的解碼 FLOPs,便保留 LLaDA-V 99.6% 的評分。它亦適用於其他基於 LLaDA 的擴散視覺語言模型,並可與視覺 token 剪枝及 KV cache 配合。

正文

Published on Oct 27

Authors:

,

,

Abstract

LOWN adaptively skips image processing during diffusion vision-language model decoding when predictions agree, reducing computation while preserving accuracy.

Diffusion vision-language models (DVLMs) process their entire input at every denoising step. Since visual tokens make up 89-97% of this input on the benchmarks we study, most of the decoding computation is spent repeatedly processing the same image. Visual token pruning methods partially relieve this burden by removing part of the visual tokens, but they lose the removed information and still process the remaining image at every step. To measure how often the image is actually needed, we compare LLaDA-V's predictions with and without the image at every decoding step and find that removing the image changes a token being unmasked at only 22.8% of steps on average and at as few as 7.9%. We therefore propose LOWN, a decoding method that decides at every step whether to process the image and how many tokens to unmask. LOWN predicts the masked tokens without the image and processes the image only when this prediction disagrees with the last image-conditioned prediction. Agreement certifies the earlier prediction for reuse, and the number of certified tokens sets how many are unmasked in that step. We further observe that the need for the image differs across benchmarks, so LOWN's adaptive decision is needed rather than a fixed schedule. Without training or a tuned threshold, LOWN retains 99.6% of LLaDA-V's score on ten benchmarks at 35.3% of the decoding FLOPs on average. LOWN also generalizes across LLaDA-based DVLMs, composes orthogonally with visual token pruning, and runs under key-value caching for further efficiency gains. Code and project page are available at https://hyeon9698.github.io/LOWN/

View arXiv page View PDF Project page GitHub 0 Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.36671 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.36671 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.36671 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co