跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-15AI 評分50

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models

AI 導讀

一項新基準測試揭示,多模態檢索模型普遍存在屬性與物件綁定失誤,涵蓋 CLIP 式雙編碼器及新近的 MLLM 嵌入模型。研究團隊推出 SABER,評估圖像與文字之間四種配對,場景最多包含十個物件;只用合成屬性綁定文字微調 MLLM 嵌入模型,無需圖像監督,仍能提升跨模態及圖像對圖像檢索,並在 SABER 上取得 SOTA 成績。

正文

Published on Oct 15

·

Submitted by

Samuel

on Oct 19

Authors:

,

,

,

,

,

Abstract

A new benchmark reveals widespread attribute-object binding failures in multimodal retrievers, and fine-tuning on synthetic text improves cross-modal binding without image supervision.

Retrieval is a fundamental task in multimodal systems: given an image or text query, the goal is to find the most relevant images or texts in a collection, often using a shared embedding space. Reliable retrieval requires these embeddings to capture not only which concepts are present, but also how they are composed. One key compositional challenge is attribute--object binding: encoding which attributes belong to which objects. Existing benchmarks provide a limited view of this capability, typically covering a subset of modality pairings, simple two-object swaps, or small attribute sets. Consequently, it remains unclear how robustly current models preserve attribute--object bindings across different modality pairings and more complex settings. We therefore introduce SABER, a controlled benchmark for evaluating attribute--object binding across all four image and text modality pairings, spanning diverse attribute types and scenes with up to ten objects. Evaluating models on SABER reveals substantial binding failures in both CLIP-style dual encoders and recent MLLM-based embedders. These findings motivate us to ask how binding can be improved efficiently. While multimodal supervision may be effective, collecting suitable images is difficult and costly. Text, on the other hand, is well suited to binding supervision: it is cheap to generate, supports a broader range of attributes and enables precisely controlled hard negatives. We thus finetune an MLLM-based embedder exclusively on synthetic attribute-binding text. Despite the absence of image supervision, the gains transfer to cross-modal and image--image retrieval while preserving general multimodal embedding performance. Our model achieves SOTA results on SABER while also improving on negation and instance-retrieval benchmarks, demonstrating broader benefits of our binding-focused supervision.

View arXiv page View PDF Project page GitHub 1 Add to collection

Get this paper in your agent:

hf papers read 2610.22602

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.22602 in a model README.md to link it from this page.

Datasets citing this paper 2

NetanelTamir/Stuck-On-Shuffle-SABER

Viewer •

Updated 40 minutes ago

•

9k

•

13

•

1

NetanelTamir/Stuck-On-Shuffle-Finetuning-Dataset

Viewer •

Updated 38 minutes ago

•

100k

•

4

•

1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.22602 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co