跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-18AI 評分42

Deep Fusion Memory:面向大語言模型的區塊級參數化記憶

Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models

AI 導讀

Deep Fusion Memory(DFM)是一種區塊級參數化記憶架構,透過閘控交叉注意力將預測的記憶融入中間隱藏狀態,推理時無需明確檢索。與現有參數化記憶方法相比,DFM 建構訓練資料所需的檢索查詢減少 32 倍、檢索項目減少 2048 倍。在六項問答任務中,DFM 相較最強基線,在 Mistral-7B-v0.3 和 Qwen3-8B-Base 上分別提升 12.71% 和 11.19%。

正文

Published on Oct 18

Authors:

,

,

,

,

,

,

Abstract

Deep Fusion Memory integrates predicted parametric memory into intermediate hidden states via gated cross-attention, reducing retrieval overhead and improving knowledge-intensive tasks without online retrieval.

Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level interpolation provides only shallow fusion. We introduce Deep Fusion Memory (DFM), a parametric memory architecture that reduces this construction workload and incorporates learned memory into intermediate hidden states. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end with the backbone frozen, and inference requires no explicit retrieval. Compared with existing parametric memory approaches, which often construct token-level memory targets, DFM requires 32times fewer retrieval queries and 2048times fewer retrieved items to construct its training data. Across six QA tasks, DFM performs better over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation yields average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.25100

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.25100 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.25100 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.25100 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co