跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 18 天前AI 評分41

DGP 如何透過跨越所有 ViT 與 LLM 層的動態閘控路徑提升 MLLM 效能?

Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers

AI 導讀

DGP(Dynamic Gated Pathways)在每個 ViT 層與 LLM 層之間建立全連接路徑,並由各 LLM 層按自身語義狀態動態選擇視覺層。在 LLaVA-1.5-7B 基線上,DGP 在 SQA^I、MM-Vet、VizWiz 和 LLaVA^W 分別提升 +4.2、+3.3、+3.1 和 +3.1。

正文

Published on Nov 4

Authors:

,

,

,

,

,

,

,

,

Abstract

Dynamic Gated Pathways connects every vision transformer layer to every large language model layer and lets each LLM layer dynamically gate its preferred visual granularity, yielding consistent multimodal gains over LLaVA-1.5 baselines.

A central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQA^{I}, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVA^{W}) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. The code is available: https://github.com/YingyingYan1/DGP.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2611.06844

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 2

Yingying11/DGP-13B

14B •

Updated about 4 hours ago

Yingying11/DGP-7B

7B •

Updated about 4 hours ago

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.06844 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.06844 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co