Can Scale Save Us From Plasticity Loss in Large Language Models?

• URL: http://arxiv.org/abs/2606.24752v1
• Date: Tue, 23 Jun 2026 16:14:52 GMT
• Title: Can Scale Save Us From Plasticity Loss in Large Language Models?
• Abstract summary: We study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem.<n>We find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size.
• Score: 7.848130688986076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer language models trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.

Related papers

• Vision Transformers that Never Stop Learning [13.804234595369058]
  We present a systematic investigation of loss of plasticity in Vision Transformers (ViTs)<n>Our analysis reveals that stacked attention modules exhibit increasing instability that exacerbates plasticity loss, while feed-forward network modules suffer even more pronounced degradation.<n>We propose ARROW, a geometry-aware model that preserves plasticity by adaptively reshaping directions using an online curvature estimate for the attention module.
  arXiv  Detail & Related papers  (2026-03-08T20:07:43Z)
• Do Neural Networks Lose Plasticity in a Gradually Changing World? [21.869234048975073]
  Loss of plasticity refers to neural networks gradually losing the ability to learn new tasks.<n>We investigate a gradually changing environment and simulate this by input/output and task sampling.<n>We show that the loss of plasticity is an artifact of abrupt tasks changes in the environment and can be largely mitigated if the world changes gradually.
  arXiv  Detail & Related papers  (2026-02-09T22:01:50Z)
• Plasticity Loss in Deep Reinforcement Learning: A Survey [15.525552360867367]
  plasticity is crucial for deep Reinforcement Learning (RL) agents. Once plasticity is lost, an agent's performance will plateau because it cannot improve its policy to account for changes in the data distribution. Loss of plasticity can be connected to many other issues plaguing deep RL, such as training instabilities, scaling failures, overestimation bias, and insufficient exploration.
  arXiv  Detail & Related papers  (2024-11-07T16:13:54Z)
• Disentangling the Causes of Plasticity Loss in Neural Networks [55.23250269007988]
  We show that loss of plasticity can be decomposed into multiple independent mechanisms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks.
  arXiv  Detail & Related papers  (2024-02-29T00:02:33Z)
• Maintaining Plasticity in Deep Continual Learning [12.27972591521307]
  We provide demonstrations of loss of plasticity using datasets repurposed for continual learning as sequences of tasks. In ImageNet, binary classification performance dropped from 89% accuracy on an early task down to 77%. New algorithm -- continual backpropagation -- modifies conventional backpropagation to re-use less-used units after each example.
  arXiv  Detail & Related papers  (2023-06-23T23:19:21Z)
• PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning [54.409634256153154]
  In Reinforcement Learning (RL), enhancing sample efficiency is crucial. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects.
  arXiv  Detail & Related papers  (2023-06-19T06:14:51Z)
• The Curse of Recursion: Training on Generated Data Makes Models Forget [70.02793975243212]
  Large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear.
  arXiv  Detail & Related papers  (2023-05-27T15:10:41Z)
• Meta-Learning Online Adaptation of Language Models [88.8947656843812]
  Large language models encode impressively broad world knowledge in their parameters. However, the knowledge in static language models falls out of date, limiting the model's effective "shelf life"
  arXiv  Detail & Related papers  (2023-05-24T11:56:20Z)
• A Survey of Large Language Models [81.06947636926638]
  Language modeling has been widely studied for language understanding and generation in the past two decades. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size.
  arXiv  Detail & Related papers  (2023-03-31T17:28:46Z)
• Continual Learning with Transformers for Image Classification [12.028617058465333]
  In computer vision, neural network models struggle to continually learn new concepts without forgetting what has been learnt in the past. We develop a solution called Adaptive Distillation of Adapters (ADA), which is developed to perform continual learning. We empirically demonstrate on different classification tasks that this method maintains a good predictive performance without retraining the model.
  arXiv  Detail & Related papers  (2022-06-28T15:30:10Z)
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models [648.3665819567409]
  Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Big-bench consists of 204 tasks, contributed by 450 authors across 132 institutions. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench.
  arXiv  Detail & Related papers  (2022-06-09T17:05:34Z)
• Entropy-based Stability-Plasticity for Lifelong Learning [17.40355682488805]
  We propose Entropy-based Stability-Plasticity (ESP) to address the stability-plasticity dilemma in neural networks. Our approach can decide dynamically how much each model layer should be modified via a plasticity factor. In some cases, it is possible to freeze layers during training leading to speed up in training.
  arXiv  Detail & Related papers  (2022-04-18T22:58:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.