跳到正文
Runway:News(網頁)·· 2025-09-24AI 評分38

A2D-VL:將自回歸視覺語言模型改編為擴散模型

Autoregressive-to-Diffusion Vision Language Models

AI 導讀

研究提出 A2D-VL 7B,透過微調 Qwen2.5-VL,將自回歸視覺語言模型改編為可並行生成的擴散模型。方法逐步增加預測窗口至 8 tokens,並採用區塊大小退火和噪聲水平退火,以保留基礎模型的能力。

正文

We adapt powerful pretrained VLMs for parallel diffusion decoding

We present a novel diffusion VLM, A2D-VL 7B (Autoregressive-to-Diffusion) for parallel generation by finetuning an existing autoregressive VLM, Qwen2.5-VL [7], on the diffusion language modeling task [8]. In particular, we adopt the masked diffusion framework [10, 11, 12] which "noises" tokens by masking them and "denoises" tokens by predicting the original tokens. We propose novel adaptation techniques (Fig. 2) that gradually increase the task difficulty during finetuning to smoothly transition from sequential to parallel decoding while preserving the base model's capabilities.

Fig 2: Our proposed diffusion vision language model, Autoregressive-to-Diffusion VLM (A2D-VL). We adapt pretrained autoregressive VLMs for parallel diffusion decoding by progressively increasing the length of the prediction window.

Further, we present novel adaptation techniques for finetuning autoregressive models into diffusion models while retaining the base model's core capabilities:

Block size annealing. The block diffusion framework enables interpolation between autoregressive and diffusion modeling: when blocks contain single tokens, we recover sequential decoding. We leverage this by gradually increasing the diffusion prediction window throughout finetuning, starting from smaller blocks and progressing to our target size of 8 tokens. This gradual progression prevents the aggressive parameter updates that would otherwise erase the base model's capabilities.

Noise level annealing. Within each token block, we apply position-dependent masking to gradually transition from easier to harder prediction tasks. Early in training, we mask the left-most tokens closest to the context more frequently (since they're easier to predict) and right-most tokens less frequently (since they're harder to predict). As training progresses, masking becomes uniform across positions, enabling any-order parallel generation within each block.

We ablate these strategies in Fig. 3, which shows the importance of both strategies for preserving benchmark performance of the base model. Concurrently, [14] also adapts Qwen2.5 for block diffusion decoding in NLP-only tasks. In contrast, we explore vision language models and propose novel adaptation techniques that are critical for retaining model capabilities.

Ablation study

Fig 3: Ablation on proposed adaptation techniques from finetuning Qwen2.5 VL 7B on 50K examples.

References

[1] Z. You, S. Nie, X. Zhang, J. Hu, J. Zhou, Z. Lu, J.R. Wen, and C. Li, "Llada-v: Large language diffusion models with visual instruction tuning," arXiv preprint arXiv:2505.16933, 2025.
[2] S. Li, K. Kallidromitis, H. Bansal, A. Gokul, Y. Kato, K. Kozuka, J. Kuen, Z. Lin, K.-W. Chang, and A. Grover, "Lavida: A large diffusion language model for multi-modal understanding," arXiv preprint arXiv:2505.16839, 2025.
[3] L. Yang, Y. Tian, B. Li, X. Zhang, K. Shen, Y. Tong, and M. Wang, "Mmada: Multimodal large diffusion language models," arXiv preprint arXiv:2505.15809, 2025.
[4] S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen, and C. Li, "Large language diffusion models," arXiv preprint arXiv:2502.09992, 2025.
[5] J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, L. Kong. "Dream 7B: Diffusion Large Language Models." arXiv preprint arXiv:2508.15487 (2025).
[6] S. Nie, F. Zhu, C. Du, T. Pang, Q. Liu, G. Zeng, M. Lin, and C. Li. "Scaling up masked diffusion models on text." ICLR 2025.
[7] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
[8] M. Arriola, A. Gokaslan, J.T. Chiu, Z. Yang, Z. Qi, J. Han, S.S. Sahoo, and V. Kuleshov. "Block diffusion: Interpolating between autoregressive and diffusion language models." ICLR 2025.
[9] C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie. "Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding." arXiv preprint arXiv:2505.22618(2025).
[10] S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. Chiu, A. Rush, and V. Kuleshov. "Simple and effective masked diffusion language models." Advances in Neural Information Processing Systems 37 (2024): 130136-130184.
[11] J. Shi, K. Han, Z. Wang, A. Doucet, and M. Titsias. "Simplified and generalized masked diffusion for discrete data." Advances in neural information processing systems 37 (2024): 103131-103167.
[12] J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li. "Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data." In The Thirteenth International Conference on Learning Representations.
[13] J. Guo, T. Zheng, Y. Bai, B. Li, Y. Wang, K. Zhu, Y. Li, G. Neubig, W. Chen, and X. Yue. "Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale." arXiv preprint arXiv:2412.05237 (2024)
[14] C. Wu, H. Zhang, S. Xue, S. Diao, Y. Fu, Z. Liu, P. Molchanov, P. Luo, S. Han, E. Xie. Fast-dLLM v2: Efficient Block-Diffusion Large Language Model. Blog post. 2025
[15] H. Dong, J. Li, B. Wu, J. Wang, Y. Zhang, and H. Guo. "Benchmarking and improving detail image caption." arXiv preprint arXiv:2405.19092 (2024).

BibTeX

@misc

{

arriola2025ar2d,
    title=

{

Adapting Autoregressive Vision Language Models for Parallel Diffusion Decoding

}

,
    author=

{

Arriola, Marianne and Venkat, Naveen and Granskog, Jonathan and Germanidis, Anastasis

}

,
    year=

{

2025

}

,
    url=

{

https://runway.com/research/autoregressive-to-diffusion-vlms

}

,
}

來源:Runway:News(網頁) · runway.com