Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2504.08000v1
• Date: Wed, 09 Apr 2025 05:43:30 GMT
• Title: Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learning
• Authors: Jiahua Lan, Sen Zhang, Haixia Pan, Ruijun Liu, Li Shen, Dacheng Tao,
• Abstract summary: We propose Neuron-level Balance between Stability and Plasticity (NBSP) method.<n>N BSP takes inspiration from the observation that specific neurons are strongly relevant to task-relevant skills.<n>N BSP significantly outperforms existing approaches in balancing stability and plasticity.
• Score: 47.023972617451044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In contrast to the human ability to continuously acquire knowledge, agents struggle with the stability-plasticity dilemma in deep reinforcement learning (DRL), which refers to the trade-off between retaining existing skills (stability) and learning new knowledge (plasticity). Current methods focus on balancing these two aspects at the network level, lacking sufficient differentiation and fine-grained control of individual neurons. To overcome this limitation, we propose Neuron-level Balance between Stability and Plasticity (NBSP) method, by taking inspiration from the observation that specific neurons are strongly relevant to task-relevant skills. Specifically, NBSP first (1) defines and identifies RL skill neurons that are crucial for knowledge retention through a goal-oriented method, and then (2) introduces a framework by employing gradient masking and experience replay techniques targeting these neurons to preserve the encoded existing skills while enabling adaptation to new tasks. Numerous experimental results on the Meta-World and Atari benchmarks demonstrate that NBSP significantly outperforms existing approaches in balancing stability and plasticity.

Related papers

• Overcoming catastrophic forgetting in neural networks [0.0]
  Catastrophic forgetting is the primary challenge that hinders continual learning.<n> Elastic Weight Consolidation is a regularization-based approach inspired by synaptic consolidation in biological neural systems.<n>Our results confirm what was shown in previous research, showing that EWC significantly reduces forgetting compared to naive training.
  arXiv  Detail & Related papers  (2025-07-14T17:04:05Z)
• When Does Neuroevolution Outcompete Reinforcement Learning in Transfer Learning Tasks? [7.479827648985631]
  Neuroevolution (NE) has recently gained attention for its robustness, scalability, and capacity to escape local optima.<n>We introduce two benchmarks: a) in stepping gates, neural networks are tasked with emulating logic circuits, with designs that emphasize modular repetition and variation b) ecorobot extends the Brax physics engine with objects such as walls and obstacles.<n>Our empirical analysis shows that NE methods vary in their transfer abilities and frequently outperform RL baselines.
  arXiv  Detail & Related papers  (2025-05-28T13:15:18Z)
• Stochastic Engrams for Efficient Continual Learning with Binarized Neural Networks [4.014396794141682]
  We propose a novel approach that integrates plasticityally-activated engrams as a gating mechanism for metaplastic binarized neural networks (mBNNs) Our findings demonstrate (A) an improved stability vs. a trade-off, (B) a reduced memory intensiveness, and (C) an enhanced performance in binarized architectures.
  arXiv  Detail & Related papers  (2025-03-27T12:21:00Z)
• Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration [28.75006029656076]
  In this paper, we introduce SSDE, a novel structure-based approach that enhances plasticity through a fine-grained allocation strategy.<n>SSDE decomposes the parameter space into forward-transfer (frozen) parameters and task-specific (trainable) parameters.<n>Experiments on the CW10-v1 Continual World benchmark demonstrate that SSDE achieves state-of-the-art performance, reaching a success rate of 95%.
  arXiv  Detail & Related papers  (2025-03-07T08:58:07Z)
• Neuroplastic Expansion in Deep Reinforcement Learning [9.297543779239826]
  Loss of plasticity in learning agents significantly impedes learning and adaptation in reinforcement learning.<n>We propose a novel approach, it Neuroplastic Expansion (NE), inspired by cortical expansion in cognitive science.<n>NE maintains learnability and adaptability throughout the entire training process by dynamically growing the network from a smaller initial size to its full dimension.
  arXiv  Detail & Related papers  (2024-10-10T14:51:14Z)
• Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
  We propose a biologically-informed framework for enhancing artificial neural networks (ANNs) Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors. We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
  arXiv  Detail & Related papers  (2024-07-05T14:11:28Z)
• Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence [59.11038175596807]
  Continual learning aims to empower artificial intelligence with strong adaptability to the real world. Existing advances mainly focus on preserving memory stability to overcome catastrophic forgetting. We propose a generic approach that appropriately attenuates old memories in parameter distributions to improve learning plasticity.
  arXiv  Detail & Related papers  (2023-08-29T02:43:58Z)
• Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning [23.15206507040553]
  We propose Auxiliary Network Continual Learning (ANCL) to equip the neural network with the ability to learn the current task. ANCL applies an additional auxiliary network which promotes plasticity to the continually learned model which mainly focuses on stability. More concretely, the proposed framework materializes in a regularizer that naturally interpolates between plasticity and stability.
  arXiv  Detail & Related papers  (2023-03-16T17:00:42Z)
• Learning threshold neurons via the "edge of stability" [33.64379851307296]
  Existing analyses of neural network training often operate under the unrealistic assumption of an extremely small learning rate. "Edge of stability" or "unstable dynamics" works on two-layer neural networks. This paper performs a detailed analysis of gradient descent for simplified models of two-layer neural networks.
  arXiv  Detail & Related papers  (2022-12-14T19:27:03Z)
• Balancing Stability and Plasticity through Advanced Null Space in Continual Learning [77.94570903726856]
  We propose a new continual learning approach, Advanced Null Space (AdNS), to balance the stability and plasticity without storing any old data of previous tasks. We also present a simple but effective method, intra-task distillation, to improve the performance of the current task. Experimental results show that the proposed method can achieve better performance compared to state-of-the-art continual learning approaches.
  arXiv  Detail & Related papers  (2022-07-25T11:04:22Z)
• Learning Dynamics and Generalization in Reinforcement Learning [59.530058000689884]
  We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training. We show that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly networks and gradient networks trained with policy methods.
  arXiv  Detail & Related papers  (2022-06-05T08:49:16Z)
• SpikePropamine: Differentiable Plasticity in Spiking Neural Networks [0.0]
  We introduce a framework for learning the dynamics of synaptic plasticity and neuromodulated synaptic plasticity in Spiking Neural Networks (SNNs) We show that SNNs augmented with differentiable plasticity are sufficient for solving a set of challenging temporal learning tasks. These networks are also shown to be capable of producing locomotion on a high-dimensional robotic learning task.
  arXiv  Detail & Related papers  (2021-06-04T19:29:07Z)
• Understanding the Role of Training Regimes in Continual Learning [51.32945003239048]
  Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. We study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima.
  arXiv  Detail & Related papers  (2020-06-12T06:00:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.