AltNet: Addressing the Plasticity-Stability Dilemma in Reinforcement Learning
- URL: http://arxiv.org/abs/2512.01034v1
- Date: Sun, 30 Nov 2025 19:02:20 GMT
- Title: AltNet: Addressing the Plasticity-Stability Dilemma in Reinforcement Learning
- Authors: Mansi Maheshwari, John C. Raisbeck, Bruno Castro da Silva,
- Abstract summary: We introduce AltNet, a reset-based approach that restores plasticity without performance degradation by leveraging twin networks.<n>We demonstrate these advantages in several high-dimensional control tasks from the DeepMind Control Suite.
- Score: 2.1487266204344473
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural networks have shown remarkable success in supervised learning when trained on a single task using a fixed dataset. However, when neural networks are trained on a reinforcement learning task, their ability to continue learning from new experiences declines over time. This decline in learning ability is known as plasticity loss. To restore plasticity, prior work has explored periodically resetting the parameters of the learning network, a strategy that often improves overall performance. However, such resets come at the cost of a temporary drop in performance, which can be dangerous in real-world settings. To overcome this instability, we introduce AltNet, a reset-based approach that restores plasticity without performance degradation by leveraging twin networks. The use of twin networks anchors performance during resets through a mechanism that allows networks to periodically alternate roles: one network learns as it acts in the environment, while the other learns off-policy from the active network's interactions and a replay buffer. At fixed intervals, the active network is reset and the passive network, having learned from prior experiences, becomes the new active network. AltNet restores plasticity, improving sample efficiency and achieving higher performance, while avoiding performance drops that pose risks in safety-critical settings. We demonstrate these advantages in several high-dimensional control tasks from the DeepMind Control Suite, where AltNet outperforms various relevant baseline methods, as well as state-of-the-art reset-based techniques.
Related papers
- ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling [57.91760520589592]
Scaling network depth has been a central driver behind the success of modern foundation models.<n>This paper revisits the default mechanism for deepening neural networks, namely residual connections.<n>We introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data.
arXiv Detail & Related papers (2026-02-09T18:54:18Z) - Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning [57.3885832382455]
We show that introducing static network sparsity alone can unlock further scaling potential beyond dense counterparts with state-of-the-art architectures.<n>Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity.
arXiv Detail & Related papers (2025-06-20T17:54:24Z) - Disentangling the Causes of Plasticity Loss in Neural Networks [55.23250269007988]
We show that loss of plasticity can be decomposed into multiple independent mechanisms.
We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks.
arXiv Detail & Related papers (2024-02-29T00:02:33Z) - SynA-ResNet: Spike-driven ResNet Achieved through OR Residual Connection [10.702093960098104]
Spiking Neural Networks (SNNs) have garnered substantial attention in brain-like computing for their biological fidelity and the capacity to execute energy-efficient spike-driven operations.
We propose a novel training paradigm that first accumulates a large amount of redundant information through OR Residual Connection (ORRC)
We then filters out the redundant information using the Synergistic Attention (SynA) module, which promotes feature extraction in the backbone while suppressing the influence of noise and useless features in the shortcuts.
arXiv Detail & Related papers (2023-11-11T13:36:27Z) - Effect of Choosing Loss Function when Using T-batching for
Representation Learning on Dynamic Networks [0.0]
T-batching is a valuable technique for training dynamic network models.
We have identified a limitation in the training loss function used with t-batching.
We propose two alternative loss functions that overcome these issues, resulting in enhanced training performance.
arXiv Detail & Related papers (2023-08-13T23:34:36Z) - Maintaining Plasticity in Deep Continual Learning [12.27972591521307]
We provide demonstrations of loss of plasticity using datasets repurposed for continual learning as sequences of tasks.
In ImageNet, binary classification performance dropped from 89% accuracy on an early task down to 77%.
New algorithm -- continual backpropagation -- modifies conventional backpropagation to re-use less-used units after each example.
arXiv Detail & Related papers (2023-06-23T23:19:21Z) - Stimulative Training++: Go Beyond The Performance Limits of Residual
Networks [91.5381301894899]
Residual networks have shown great success and become indispensable in recent deep neural network models.
Previous research has suggested that residual networks can be considered as ensembles of shallow networks.
We identify a problem that is analogous to social loafing, whereworks within a residual network are prone to exert less effort when working as part of a group compared to working alone.
arXiv Detail & Related papers (2023-05-04T02:38:11Z) - Stimulative Training of Residual Networks: A Social Psychology
Perspective of Loafing [86.69698062642055]
Residual networks have shown great success and become indispensable in today's deep models.
We aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing.
We propose a new training strategy to strengthen the performance of residual networks.
arXiv Detail & Related papers (2022-10-09T03:15:51Z) - Learning Fast and Slow for Online Time Series Forecasting [76.50127663309604]
Fast and Slow learning Networks (FSNet) is a holistic framework for online time-series forecasting.
FSNet balances fast adaptation to recent changes and retrieving similar old knowledge.
Our code will be made publicly available.
arXiv Detail & Related papers (2022-02-23T18:23:07Z) - Iterative Network Pruning with Uncertainty Regularization for Lifelong
Sentiment Classification [25.13885692629219]
Lifelong learning is non-trivial for deep neural networks.
We propose a novel iterative network pruning with uncertainty regularization method for lifelong sentiment classification.
arXiv Detail & Related papers (2021-06-21T15:34:13Z) - Bridging the Gap Between Target Networks and Functional Regularization [61.051716530459586]
We show that Target Networks act as an implicit regularizer which can be beneficial in some cases, but also have disadvantages.
We propose an explicit Functional Regularization alternative that is flexible and a convex regularizer in function space.
Our findings emphasize that Functional Regularization can be used as a drop-in replacement for Target Networks and result in performance improvement.
arXiv Detail & Related papers (2021-06-04T17:21:07Z) - Network Pruning That Matters: A Case Study on Retraining Variants [11.503165599245467]
We study the effective of different retraining mechanisms while doing pruning.
We demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks.
arXiv Detail & Related papers (2021-05-07T12:03:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.