Towards Better Plasticity-Stability Trade-off in Incremental Learning: A simple Linear Connector

• URL: http://arxiv.org/abs/2110.07905v1
• Date: Fri, 15 Oct 2021 07:37:20 GMT
• Title: Towards Better Plasticity-Stability Trade-off in Incremental Learning: A simple Linear Connector
• Authors: Guoliang Lin, Hanglu Chu, Hanjiang Lai
• Abstract summary: Plasticity-stability dilemma is a main problem for incremental learning. We show that a simple averaging of two independently optimized optima of networks, null-space projection for past tasks and simple SGD for the current task, can attain a meaningful balance between preserving already learned knowledge and granting sufficient flexibility for learning a new task.
• Score: 8.13916229438606
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Plasticity-stability dilemma is a main problem for incremental learning, with plasticity referring to the ability to learn new knowledge, and stability retaining the knowledge of previous tasks. Due to the lack of training samples from previous tasks, it is hard to balance the plasticity and stability. For example, the recent null-space projection methods (e.g., Adam-NSCL) have shown promising performance on preserving previous knowledge, while such strong projection also causes the performance degradation of the current task. To achieve better plasticity-stability trade-off, in this paper, we show that a simple averaging of two independently optimized optima of networks, null-space projection for past tasks and simple SGD for the current task, can attain a meaningful balance between preserving already learned knowledge and granting sufficient flexibility for learning a new task. This simple linear connector also provides us a new perspective and technology to control the trade-off between plasticity and stability. We evaluate the proposed method on several benchmark datasets. The results indicate our simple method can achieve notable improvement, and perform well on both the past and current tasks. In short, our method is an extremely simple approach and achieves a better balance model.

Related papers

• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learning [33.560003528712414]
  Self-supervised learning (SSL) has emerged as an effective paradigm for deriving general representations from vast amounts of unlabeled data. This poses a challenge in striking a balance between stability and plasticity when adapting to new information. We propose Branch-tuning, an efficient and straightforward method that achieves a balance between stability and plasticity in continual SSL.
  arXiv  Detail & Related papers  (2024-03-27T05:38:48Z)
• Towards Plastic and Stable Exemplar-Free Incremental Learning: A Dual-Learner Framework with Cumulative Parameter Averaging [12.168402195820649]
  We propose a Dual-Learner framework with Cumulative. Averaging (DLCPA) We show that DLCPA outperforms several state-of-the-art exemplar-free baselines in both Task-IL and Class-IL settings.
  arXiv  Detail & Related papers  (2023-10-28T08:48:44Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• On the Stability-Plasticity Dilemma of Class-Incremental Learning [50.863180812727244]
  A primary goal of class-incremental learning is to strike a balance between stability and plasticity. This paper aims to shed light on how effectively recent class-incremental learning algorithms address the stability-plasticity trade-off.
  arXiv  Detail & Related papers  (2023-04-04T09:34:14Z)
• Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning [23.15206507040553]
  We propose Auxiliary Network Continual Learning (ANCL) to equip the neural network with the ability to learn the current task. ANCL applies an additional auxiliary network which promotes plasticity to the continually learned model which mainly focuses on stability. More concretely, the proposed framework materializes in a regularizer that naturally interpolates between plasticity and stability.
  arXiv  Detail & Related papers  (2023-03-16T17:00:42Z)
• New Insights on Relieving Task-Recency Bias for Online Class Incremental Learning [37.888061221999294]
  In all settings, the online class incremental learning (OCIL) is more challenging and can be encountered more frequently in real world. To strike a preferable trade-off between stability and plasticity, we propose an Adaptive Focus Shifting algorithm.
  arXiv  Detail & Related papers  (2023-02-16T11:52:00Z)
• Balancing Stability and Plasticity through Advanced Null Space in Continual Learning [77.94570903726856]
  We propose a new continual learning approach, Advanced Null Space (AdNS), to balance the stability and plasticity without storing any old data of previous tasks. We also present a simple but effective method, intra-task distillation, to improve the performance of the current task. Experimental results show that the proposed method can achieve better performance compared to state-of-the-art continual learning approaches.
  arXiv  Detail & Related papers  (2022-07-25T11:04:22Z)
• Efficient Empowerment Estimation for Unsupervised Stabilization [75.32013242448151]
  empowerment principle enables unsupervised stabilization of dynamical systems at upright positions. We propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel. We show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from images.
  arXiv  Detail & Related papers  (2020-07-14T21:10:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.