Zyphra Research 發現 GPT 式 Transformer 在持續及平穩學習中出現可塑性損失

ResearchPlasticity Loss in Continual LearningZyphra Research studies the loss of the ability to learn after continued training—loss of plasticity—in GPT-style decoder-only Transformers. We discover that models with 5M to 314M non-embedding parameters lose plasticity when trained on a multilingual continual learning problem, as measured by deterioration on a held-out probing task.We derive a scaling law predicting that the onset of plasticity loss scales sublinearly with the number of non-embedding parameters. This prediction implies that scaling the number of parameters to prevent plasticity loss has diminishing returns and may be an inefficient approach to addressing the problem. In addition, we demonstrate that plasticity loss also occurs during stationary learning, suggesting that current approaches to training large language models are similarly susceptible to plasticity loss. Our work indicates that loss of plasticity might be a key bottleneck for full continual learning of LLMs.

來源：Zyphra Research · zyphra.com