跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-13AI 評分50

PORTAL 重新檢視多模態 LLM 投影器訓練,提出免配對數據初始化方法

Rethinking Projector Training in Multimodal LLMs

AI 導讀

論文提出 PORTAL,透過最優傳輸及各模態摘要統計,在無需配對圖像描述數據或梯度最佳化的情況下初始化多模態 LLM 投影器,省去投影器訓練階段。研究亦發現,投影器只用 10% 至 20% 訓練數據已能恢復大部分下游效能提升,不同數據集訓練的投影器亦可線性插值並大致保留效能。

正文

Published on Oct 13

Authors:

,

,

Abstract

PORTAL initializes multimodal projectors without paired data by aligning vision and language feature distributions via optimal transport, matching standard training performance while eliminating projector training.

Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood. We find that (i) using only 10 to 20% of the data for training the projector already recovers most downstream performance gains, and (ii) projectors trained on different datasets can be linearly interpolated while largely preserving performance. These results suggest that projector training mainly performs a coarse alignment, which many different projectors achieve equally well. Motivated by these findings, we propose PORTAL, a training-free projector initialization that requires neither paired image-caption data nor gradient-based optimization, but only per-modality summary statistics. PORTAL computes an optimal-transport-based map between Gaussian approximations of the vision and language feature distributions on a shared principal subspace, so that the projector training stage can be skipped entirely. Without any paired data, PORTAL matches the standard pipeline trained on 558k image-caption pairs across six LLM backbones, two vision encoders, and 16 benchmarks. It outperforms that pipeline in 3 of 7 (vision encoder, LLM) settings and reaches an average 98% of its score across the other four.

View arXiv page View PDF Project page GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2610.18598

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.18598 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.18598 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.18598 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co