跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-08AI 評分43

Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers

AI 導讀

研究發現,混合文字、圖片及文字圖片融合文件的檢索表現會受模態組成影響,無關文字造成的性能下降比同等數量的無關圖片更嚴重。檢索器對文字表示有模態偏好,可能令無關文字的相似度高於相關圖片。研究提出 Trident,以 Multi-Positive View InfoNCE 聯合優化相關性判別與正樣本視圖平衡,提升 CLIP-based 和 VLM-based 架構的混合模態檢索表現。

正文

Abstract

Dense retrievers have made significant progress on text and image corpora, but whether these capabilities extend reliably to mixed corpora containing text, image, and fused text-image documents remains unclear. In this paper, we systematically examine retrievers across architectures and find that their performance is highly sensitive to modality composition. As image documents are progressively replaced with semantically corresponding text representations, retrieval performance follows a pronounced V-shaped curve, remaining strong on single-modality corpora but degrading substantially when modalities coexist. In particular, irrelevant text causes more severe degradation than an equal number of irrelevant images, a phenomenon we term Chaos in the Text. Further analysis reveals modality preference, whereby text representations receive systematically higher similarity scores, allowing irrelevant text to outrank relevant images. To mitigate this bias, we introduce Trident, which constructs text, image, and fused text-image views of each document as co-equal positives and jointly optimizes relevance discrimination and positive-view balance through Multi-Positive View InfoNCE. Experiments across visual document and natural image benchmarks show that trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures, reduces sensitivity to modality composition and text distractors, and increases average single-modality retrieval performance.

View arXiv page View PDF GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.11816 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.11816 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.11816 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co