Cortex: Consolidating Context into Parametric Memory
Cortex: Consolidating Context into Parametric Memory
Cortex 透過生成前瞻性查詢,將依賴上下文的輸出蒸餾為可合併的低秩或結構化前饋網絡更新,將上下文帶來的能力增益整合至模型參數。研究以涵蓋 4B–120B 密集及混合專家模型的測試集評估整合與保留效果。在 14B 模型上連續進行 50 次事實編輯後,Cortex-Structural 恢復 88.9% 的上下文增益,同時保留 99.8% 的原有非目標表現。
Published on Oct 14
Authors:
,
,
,
,
,
,
Abstract
Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
View arXiv page View PDF Add to collection
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.20399 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.20399 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.20399 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co