Learning Multimodal Embeddings with Evidence-Aligned Readout
Learning Multimodal Embeddings with Evidence-Aligned Readout
EviAlign 將語義證據生成與邊界讀出結合於同一多模態大語言模型,把五個語義單元邊界的狀態聚合為單一正規化嵌入。受控研究顯示,一致的語義組織在證據邊界讀出下帶來 2.39 分優勢,較按長度設定訓練位置時的 0.65 分高出 1.74 分。EviAlign 使用 500K 訓練配對,在 12 項 MMEB 檢索任務取得 76.9 平均 Recall@1,並保留單向量索引及評分。
Published on Sep 27
·
on Oct 8
Authors:
,
,
,
,
,
,
,
,
,
Abstract
Multimodal large language models can expose task-relevant evidence through generation, but producing useful evidence does not by itself determine how it enters a retrieval embedding. We study whether the semantic organization of that evidence can also specify where representations are read. To address this question, we introduce EviAlign, which couples Semantic Evidence Generation with Boundary Readout in a shared multimodal large language model. It organizes evidence into five semantic units, reads the contextualized state at each unit boundary, and aggregates these states into a single normalized embedding. Generation and contrastive retrieval objectives jointly train this shared structure. With the same trailing readout, semantic evidence and free-form CoT yield nearly identical retrieval performance, suggesting that evidence organization alone does not explain the full gain. A controlled 2times3 study compares consistent and permuted evidence organization across three readout strategies, using training targets with matched evidence spans. With five readout states and the same mean pooling, the advantage of consistent semantic organization grows from 0.65 points at length-based training positions to 2.39 at evidence boundaries, yielding a 1.74-point co-design interaction. Across 12 MMEB retrieval tasks, EviAlign achieves 76.9 average Recall@1 with 500K training pairs while retaining single-vector indexing and scoring.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2609.33659
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.33659 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.33659 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.33659 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co