跳到正文
Apple Machine Learning Research·· 12 天前AI 評分45

門控的重要性:狀態空間模型中的記憶化與上下文學習

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

AI 導讀

研究以理論分析和實驗指出,門控機制會令狀態空間模型(SSMs)先學會把資訊記在權重中的「記憶化」解法,並延遲甚至阻礙模型收斂至正確的上下文學習解法,即使架構或記憶容量並無根本限制。另一方面,門控通常有助模型推廣至更長的序列,研究結果揭示門控對 SSMs 訓練動態與泛化的關鍵作用。

正文

AuthorsWilliam L. Tong†**, Aryo Lotfi, Emmanuel Abbe, Kostas Vaggelakos, Vishnu Banna, Etai Littwin, Josh Susskind, Cengiz Pehlevan†, Eran Malach

State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studying the role of the gating mechanism, a prevalent component in modern recurrent networks. Specifically, we show through theory and experiments that this gating mechanism causes SSMs to first learn an in-weights “memorization” solution, while delaying, or even preventing, convergence to a correct in-context learning solution. Importantly, this happens even in cases where there are no fundamental limitations due to the architecture or its memory capacity. On the other hand, we find that gating is often beneficial for improving generalization to long sequence lengths. Our results illuminate the crucial role of the gating mechanism in shaping both the training dynamics and generalization of SSMs, and provide a basis for understanding and improving linear-time models.

  • † Harvard University
  • ** Work done while at Apple

Related readings and updates.

State Space Models (SSMs) have become the leading alternative to Transformers for sequence modeling. Their primary advantage is efficiency in long-context and long-form generation, enabled by fixed-size memory and linear scaling of computational complexity. We begin this work by showing a simple theoretical result stating that SSMs cannot accurately solve any “truly long-form” generation problem (in a sense we formally define), undermining…

Read more

Transformers have demonstrated impressive performance on class-conditional ImageNet benchmarks, achieving state-of-the-art FID scores. However, their computational complexity increases with transformer depth/width or the number of input tokens and requires patchy approximation to operate on even latent input sequences. In this paper, we address these issues by presenting a novel approach to enhance the efficiency and scalability of image…

Read more

來源:Apple Machine Learning Research · machinelearning.apple.com