Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective

• URL: http://arxiv.org/abs/2605.09044v1
• Date: Sat, 09 May 2026 16:35:00 GMT
• Title: Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective
• Abstract summary: We take a practical view to interpret plasticity as trainability, i.e., a neural network's future optimization gain on a target task.<n>We propose a novel metric, called optimization readiness, which combines gradient strength and gradient reliability.
• Score: 41.50730100158907
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep continual learning requires models to adapt to new tasks without retraining from scratch. However, neural networks can lose their ability to adapt to new tasks after training on previous ones, a phenomenon known as loss of plasticity. There have been several explanations and diagnostics proposed for plasticity loss. Motivated by the philosophy that "all models are wrong, but some are useful", we ask: can existing diagnostics predict a neural network's plasticity? In this work, we take a practical view to interpret plasticity as trainability, i.e., a neural network's future optimization gain on a target task. We first take a theoretical approach, showing, by constructing a few counterexamples, that some widely adopted diagnostics of plasticity, including representation rank and neural tangent kernel rank, can fail to predict the loss of trainability in both regression and classification settings. We instead propose a novel metric, called optimization readiness, which combines gradient strength and gradient reliability. We prove that optimization readiness lower bounds one-step optimization gain under standard smoothness assumptions, providing a theoretical guarantee for its predictive power. Empirically, we show that across commonly used deep continual learning settings, such as Slowly-Changing Regression and Permuted MNIST, optimization readiness more reliably ranks checkpoints by trainability than prior diagnostics, even with substantially fewer samples.

Related papers

• Rethinking Plasticity in Deep Reinforcement Learning [3.18807491942654]
  This paper investigates the fundamental mechanisms driving plasticity loss in deep reinforcement learning (RL)<n>We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks.<n>Our findings provide a rigorous optimization-based framework for understanding and restoring network plasticity in complex domains.
  arXiv  Detail & Related papers  (2026-03-22T11:27:16Z)
• Simmering: Sufficient is better than optimal for training neural networks [0.0]
  We introduce simmering, a physics-based method that trains neural networks to generate weights and biases that are merely good enough''<n>We show that simmering corrects neural networks that are overfit by Adam, and show that simmering avoids overfitting if deployed from the outset.<n>Our results question optimization as a paradigm for neural network training, and leverage information-geometric arguments to point to the existence of classes of sufficient training algorithms.
  arXiv  Detail & Related papers  (2024-10-25T18:02:08Z)
• Just How Flexible are Neural Networks in Practice? [89.80474583606242]
  It is widely believed that a neural network can fit a training set containing at least as many samples as it has parameters. In practice, however, we only find solutions via our training procedure, including the gradient and regularizers, limiting flexibility.
  arXiv  Detail & Related papers  (2024-06-17T12:24:45Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatory [12.689249854199982]
  We show that the test risk of RF regression decreases monotonically with both the number of features and the number of samples. We then demonstrate that, for a large class of tasks characterized by powerlaw eigenstructure, training to near-zero training loss is obligatory.
  arXiv  Detail & Related papers  (2023-11-24T18:27:41Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths [12.068608358926317]
  First-order optimization algorithms are known to efficiently locate favorable minima in deep neural networks. We focus on the fundamental geometric properties of sampled quantities of optimization on two key paths. Our findings suggest that not only do optimization trajectories never encounter significant obstacles, but they also maintain stable dynamics during the majority of training.
  arXiv  Detail & Related papers  (2023-06-20T22:10:40Z)
• Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning [70.52689048213398]
  This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features. We find new and interesting properties that do not exist in single-task linear regression. Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
  arXiv  Detail & Related papers  (2023-04-09T20:36:13Z)
• Slimmable Networks for Contrastive Self-supervised Learning [69.9454691873866]
  Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. We introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers. A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks.
  arXiv  Detail & Related papers  (2022-09-30T15:15:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.