SIMIT:透過測試時想像提升視覺語言模型
SIMIT: Self-Improving Vision-Language Models via Imagination at Test-Time
SIMIT 是一種測試時自我改進框架,讓視覺語言模型在回答未標註查詢前生成專屬練習資料,無需外部標註或模型。它合成多樣的問題、答案與圖像組合,並以多階段驗證和難度篩選檢查及挑選樣本。SIMIT 在 17 個基準測試中勝過現有自我改進方法;最佳配置較 BAGEL-7B 平均相對提升 +7.20%,免訓練的 SIMIT-ICL 提升 +6.98%。
Published on Oct 11
Authors:
,
Abstract
SIMIT is a test-time framework that lets vision-language models generate their own practice data to improve answers on unlabeled queries without external annotations.
We introduce SIMIT, a test-time self-improvement framework in which a vision-language model creates its own query-specific training data before answering an unlabeled test query. Our goal is to improve performance on a given target test set without annotations or external models. To pursue this goal, some test-time methods improve answers through longer reasoning or repeated sampling but leave the model unchanged. Others update the model using rewards based on majority agreement among multiple candidate answers or the model's own judgments of correctness, but such rewards can reinforce existing mistakes. SIMIT instead imagines similar problems whose solutions it controls: a model need not correctly solve the target query to construct useful practice examples. It synthesizes diverse (question, answer, image) triplets by specifying each answer before realizing its image. To support diverse multimodal tasks, SIMIT combines native image generation with an extensible skill library for structured visuals, including documents, charts, and diagrams. Within this pipeline, multi-stage verification checks that images support their assigned answers, while difficulty filtering retains informative examples. Adaptive budgeting further targets this practice by allocating more samples to queries the model is less confident in answering. The resulting data can improve the model in-context (SIMIT-ICL) or in-weight (SIMIT-FT). Across 17 diverse benchmarks, SIMIT outperforms existing self-improvement methods. Its best configuration achieves a +7.20% mean relative gain over BAGEL-7B, while training-free SIMIT-ICL achieves +6.98%.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.00053
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.00053 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.00053 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.00053 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co