Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

• URL: http://arxiv.org/abs/2607.24996v1
• Date: Mon, 27 Jul 2026 18:53:00 GMT
• Title: Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
• Abstract summary: We propose Calibrated Partial Resets (CPR) to preserve plasticity without destabilizing training.<n>CPR avoids policy collapse over 400M training steps in SlipperyAnt.<n>CPR highlights utility-scaled reinitialization as a promising direction for continual learning.
• Score: 3.1681974483553663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning. Recently, neuron resets have been used to maintain gradient flow and restore plasticity. However, full unit reinitialization often sacrifices peak performance and can destabilize training, leading to policy collapse. To preserve plasticity without destabilizing training, we propose Calibrated Partial Resets (CPR), an optimizer that periodically pulls low-utility neurons toward their initialization, with pull strength scaled by each neuron's utility. Unlike binary reset methods, partial resets avoid brittleness; unlike uniform decay, calibrated utility-scaling concentrates adjustment on the units that need it most. Among compared methods, only CPR avoids policy collapse over 400M training steps in SlipperyAnt, and it outperforms prior decay and reset-based methods on Continual MetaWorld and Continual MinAtar benchmarks. Ablations reveal a tunable trade-off between plasticity and peak performance, highlighting utility-scaled reinitialization as a promising direction for continual learning.

Related papers

• PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networks [5.059907627897051]
  As objectives shift, neurons progressively fall dormant and the shared policy loses the capacity to learn.<n>We show that sustained non-stationarity damages this internal state directly.<n> PRIME improves interquartile mean return by 24.9% over MAPPO.
  arXiv  Detail & Related papers  (2026-07-20T13:19:58Z)
• RL Forgets! Towards Continual Policy Optimization [57.4336338996653]
  Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks.<n>Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting.<n>We propose Continual Policy Optimization (CPO), a replay-free framework grounded in a prior-task behavioral KL objective.
  arXiv  Detail & Related papers  (2026-07-05T15:46:11Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networks [0.0]
  Continual learning research attempts to conserve two fundamental capabilities: new knowledge acquisition and the preservation of previously acquired knowledge.<n>Recent research has identified multiple mechanisms underlying this phenomenon, including neuron saturation, norm growth, and loss of useful curvature directions.<n> Adaptive reset-based interventions have emerged as practical solutions to restore trainability.
  arXiv  Detail & Related papers  (2026-05-07T18:42:22Z)
• Stochastic Resetting Accelerates Policy Convergence in Reinforcement Learning [5.900714266080361]
  resetting is a powerful mechanism for optimizing first-passage properties.<n>We show that resetting accelerates convergence even when it does not reduce the search time of a purely diffusive agent.<n>In a continuous control task with neural-network-based value approximation, we show that random resetting improves deep reinforcement learning when exploration is difficult.
  arXiv  Detail & Related papers  (2026-03-17T17:50:32Z)
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices [61.361819972410046]
  We show why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE.<n>This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training.
  arXiv  Detail & Related papers  (2025-12-01T07:45:39Z)
• The Lifecycle Principle: Stabilizing Dynamic Neural Networks with State Memory [0.0]
  I propose a regularization mechanism centered on state memory.<n>Instead of re-initializing a revived neuron, my method restores its parameters to their last known effective state.<n> Experiments on image classification benchmarks demonstrate that my method improves generalization and robustness.
  arXiv  Detail & Related papers  (2025-08-24T05:20:01Z)
• A Simple Baseline for Stable and Plastic Neural Networks [3.2635082758250693]
  Continual learning in computer vision requires that models adapt to a continuous stream of tasks without forgetting prior knowledge.<n>We introduce RDBP, a low-overhead baseline that unites two complementary mechanisms: ReLUDown, a lightweight activation modification that preserves feature sensitivity while preventing neuron dormancy, and Decreasing Backpropagation, a biologically inspired gradient-scheduling scheme that progressively shields early layers from catastrophic updates.
  arXiv  Detail & Related papers  (2025-07-14T13:18:26Z)
• Instance-dependent Early Stopping [57.912273923450726]
  We propose an Instance-dependent Early Stopping (IES) method that adapts the early stopping mechanism from the entire training set to the instance level.<n>IES considers an instance as mastered if the second-order differences of its loss value remain within a small range around zero.<n>IES can reduce backpropagation instances by 10%-50% while maintaining or even slightly improving the test accuracy and transfer learning performance of a model.
  arXiv  Detail & Related papers  (2025-02-11T13:34:09Z)
• Forget Forgetting: Continual Learning in a World of Abundant Memory [55.64184779530581]
  Continual learning has traditionally focused on minimizing exemplar memory.<n>This paper challenges this paradigm by investigating a more realistic regime.<n>We find that the core challenge shifts from stability to plasticity, as models become biased toward prior tasks and struggle to learn new ones.
  arXiv  Detail & Related papers  (2025-02-11T05:40:52Z)
• Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages [56.98243487769916]
  Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning. We propose Adaptive RR, which dynamically adjusts the replay ratio based on the critic's plasticity level.
  arXiv  Detail & Related papers  (2023-10-11T12:05:34Z)
• When Does Re-initialization Work? [50.70297319284022]
  Re-initialization has been observed to improve generalization in recent works. It is neither widely adopted in deep learning practice nor is it often used in state-of-the-art training protocols. This raises the question of when re-initialization works, and whether it should be used together with regularization techniques.
  arXiv  Detail & Related papers  (2022-06-20T21:23:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.