The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning

• URL: http://arxiv.org/abs/2604.01913v1
• Date: Thu, 02 Apr 2026 11:29:46 GMT
• Title: The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning
• Abstract summary: Deep reinforcement learning (RL) suffers from plasticity loss severely due to the nature of non-stationarity.<n>We study the plasticity loss problem from the theoretical perspective of network optimization.<n>We propose Sample Weight Decay -- a lightweight method to restore gradient magnitude.
• Score: 62.72522261010872
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep reinforcement learning (RL) suffers from plasticity loss severely due to the nature of non-stationarity, which impairs the ability to adapt to new data and learn continually. Unfortunately, our understanding of how plasticity loss arises, dissipates, and can be dissolved remains limited to empirical findings, leaving the theoretical end underexplored.To address this gap, we study the plasticity loss problem from the theoretical perspective of network optimization. By formally characterizing the two culprit factors in online RL process: the non-stationarity of data distributions and the non-stationarity of targets induced by bootstrapping, our theory attributes the loss of plasticity to two mechanisms: the rank collapse of the Neural Tangent Kernel (NTK) Gram matrix and the $Θ(\frac{1}{k})$ decay of gradient magnitude. The first mechanism echoes prior empirical findings from the theoretical perspective and sheds light on the effects of existing methods, e.g., network reset, neuron recycle, and noise injection. Against this backdrop, we focus primarily on the second mechanism and aim to alleviate plasticity loss by addressing the gradient attenuation issue, which is orthogonal to existing methods. We propose Sample Weight Decay -- a lightweight method to restore gradient magnitude, as a general remedy to plasticity loss for deep RL methods based on experience replay. In experiments, we evaluate the efficacy of \methodName upon TD3, \myadded{Double DQN} and SAC with SimBa architecture in MuJoCo, \myadded{ALE} and DeepMind Control Suite tasks. The results demonstrate that \methodName effectively alleviates plasticity loss and consistently improves learning performance across various configurations of deep RL algorithms, UTD, network architectures, and environments, achieving SOTA performance on challenging DMC Humanoid tasks.

Related papers

• Rethinking Plasticity in Deep Reinforcement Learning [3.18807491942654]
  This paper investigates the fundamental mechanisms driving plasticity loss in deep reinforcement learning (RL)<n>We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks.<n>Our findings provide a rigorous optimization-based framework for understanding and restoring network plasticity in complex domains.
  arXiv  Detail & Related papers  (2026-03-22T11:27:16Z)
• Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning [14.196969540084929]
  We show that deep neural networks suffer from loss of plasticity in deep continual learning.<n>We introduce the notion of $tau$-trainability and show that current plasticity preserving algorithms can be unified under this framework.<n> Experiments on continual supervised and reinforcement learning tasks confirm that combining these two regularizers effectively preserves plasticity.
  arXiv  Detail & Related papers  (2025-09-26T13:28:53Z)
• Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn [22.354498355750465]
  We study the loss of plasticity in deep continual RL from the lens of churn.<n>We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix.<n>We introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance.
  arXiv  Detail & Related papers  (2025-05-31T14:58:22Z)
• Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning [122.67854581396578]
  Plasticine is an open-source framework for benchmarking plasticity optimization in deep reinforcement learning.<n>Plasticine provides single-file implementations of over 13 mitigation methods, 10 evaluation metrics, and learning scenarios.
  arXiv  Detail & Related papers  (2025-04-24T12:32:13Z)
• A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning [7.767611890997147]
  We show that plasticity loss is pervasive under domain shift in on-policy deep RL. We find that a class of regenerative'' methods are able to consistently mitigate plasticity loss in a variety of contexts.
  arXiv  Detail & Related papers  (2024-05-29T14:59:49Z)
• Disentangling the Causes of Plasticity Loss in Neural Networks [55.23250269007988]
  We show that loss of plasticity can be decomposed into multiple independent mechanisms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks.
  arXiv  Detail & Related papers  (2024-02-29T00:02:33Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages [56.98243487769916]
  Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning. We propose Adaptive RR, which dynamically adjusts the replay ratio based on the critic's plasticity level.
  arXiv  Detail & Related papers  (2023-10-11T12:05:34Z)
• PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning [54.409634256153154]
  In Reinforcement Learning (RL), enhancing sample efficiency is crucial. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects.
  arXiv  Detail & Related papers  (2023-06-19T06:14:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.