HiUni 如何在共享視覺空間中統一理解與生成
HiUni: Harmonizing Understanding and Generation in Shared Visual Space
HiUni 探索將預訓練視覺語言模型擴展為統一多模態模型,讓同一個語言對齊 ViT 和 dense Transformer 主幹同時支援視覺理解與生成,並將任務專屬計算限制在輸出端。模型以 Pixel Diffusion Head 優化像素空間生成,並透過輕量 HiGate 路徑補充合成所需的細緻空間資訊。實驗顯示,HiUni 的平均理解評分與預訓練視覺語言模型相若,圖像生成和編輯表現具競爭力。
Published on Nov 19
Authors:
,
,
,
,
,
,
,
,
,
,
,
Abstract
Unified multimodal models aim to support visual understanding and generation within a single framework. However, prevailing approaches often separate their visual representations or backbone computation to accommodate the different requirements of these two tasks, which limits the degree of unification. In this paper, we present HiUni, which revisits this design choice and explores whether a pretrained vision language model can be directly extended into a unified model with extensive parameter sharing. Specifically, HiUni reuses the same language-aligned ViT and dense Transformer backbone for both capabilities, while confining task-specific computation to the output side. Yet such a design introduces two key challenges. Generation in pixel space is difficult to optimize, and semantic abstraction in pretrained ViT discards fine spatial details required for synthesis. To address these challenges, HiUni introduces a Pixel Diffusion Head tailored to direct pixel-space optimization and a lightweight HiGate pathway that provides fine-grained spatial details near the output stage. Extensive experiments show that HiUni achieves a similar average understanding score to the pretrained VLM while attaining competitive performance on image generation and editing. Controlled comparisons and analyses further demonstrate that a single dense backbone can effectively support both capabilities without task-specific backbone computation, highlighting a simpler path toward unified multimodal modeling.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.26107
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 1
Image-Text-to-Text • 5B • Updated 1 minute ago • 1ernie-research/HiUni
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2611.26107 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.26107 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co