跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 24 天前AI 評分47

視覺語言模型在哪些 3D 推理任務上失敗?

Where Do Vision Language Models Fail at 3D Reasoning?

AI 導讀

BedBench 顯示,視覺語言模型難以準確計算 3D 物件各顏色的數量,並在回答中一致地運用這些計數。基準涵蓋 13 項任務;人類準確率為 93.3%,最佳模型為 7.2%。針對性微調令一個模型的準確率由 3.0% 升至 68.6%,但其答案仍與自身計數不一致;程式碼和數據集已公開。

正文

Published on Oct 29

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Vision-language models often judge 3D object properties correctly but fail to recover and consistently apply exact per-color counts, motivating evaluation by exact correctness, consistency, and visual dependence.

Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3% accuracy, compared with 7.2% for the best model. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but reveals a further difficulty: its answers about the most common colors remain inconsistent with its own per-color counts. Supplying those counts as text improves its judgments, showing that recovering a property does not mean using it consistently. Reasoning, scaling, and added visual cues each improve individual judgments, but none makes exact answers reliable. To locate the difficulty, we probe Qwen3.5-27B: exact per-color counts remain recoverable at the language model's input, yet the model's answers rarely recover them (65.0% for the probe vs. 3.0% for the model). Image-removal controls further show that some apparent successes require no visual evidence. These findings motivate evaluating 3D reasoning by exact correctness, consistent use, and image dependence, rather than accuracy alone. Code and dataset are publicly available: https://github.com/leduckhai/BedBench

View arXiv page View PDF Project page GitHub 0 Add to collection

Get this paper in your agent:

hf papers read 2610.39639

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.39639 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.39639 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.39639 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co