跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-10AI 評分38

通用多模態嵌入的對比訓練策略,哪些真正有效?一項受控研究

What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

AI 導讀

受控評測顯示,多數對比訓練策略相較標準對齊基線提升有限,部分策略更會降低整體表現。結合困難負例挖掘、假陰性緩解與困難批次構建,在 Qwen2-VL-2B 上令 MMEB 平均提升 6.0%,表現最佳。

正文

Published on Oct 10

Authors:

,

,

,

,

,

Abstract

Contrastive learning has become the standard paradigm for multimodal representation learning, aligning semantically matched pairs while separating mismatched ones in a shared embedding space. Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood. This motivates a fundamental question: Do sophisticated contrastive strategies consistently bring meaningful downstream gains in multimodal embedding learning? To address this, we systematically evaluate representative strategies under a unified benchmarking framework on MMEB. Surprisingly, direct evaluations reveal that most existing strategies offer marginal gains over a standard aligned baseline, with several even degrading overall performance. To uncover the underlying failure modes, we introduce an MLLM-as-a-Judge diagnostic framework that dissects top-1 retrieval behaviors into labeled positives, false negatives (unlabeled valid matches), and hard/easy negative errors. Our systematic analysis yields three key insights: (1) Gradient distortion: Upweighting gradients from high-similarity negatives consistently hurts downstream generalization; (2) Distribution illusion: Wider separation between positive and negative similarity distributions does not necessarily translate to better retrieval performance; and (3) Synergistic training: While false-negative mitigation or hard-batch construction yields limited gains in isolation, integrating hard-negative mining, false-negative mitigation, and hard-batch construction achieves optimal performance, delivering an average improvement of 6.0% on MMEB with Qwen2-VL-2B. Overall, our findings demystify popular contrastive strategies and provide actionable recipes for training universal multimodal embedding models.

View arXiv page View PDF Project page GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.15035

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

alibaba-multimodal-industrial-ai/UME-Recipes

Feature Extraction •

Updated about 1 hour ago

•

3

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.15035 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.15035 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co