Learning without Isolation: Pathway Protection for Continual Learning

• URL: http://arxiv.org/abs/2505.18568v1
• Date: Sat, 24 May 2025 07:16:55 GMT
• Title: Learning without Isolation: Pathway Protection for Continual Learning
• Authors: Zhikang Chen, Abudukelimu Wuerkaixi, Sen Cui, Haoxuan Li, Ding Li, Jingfeng Zhang, Bo Han, Gang Niu, Houfang Liu, Yi Yang, Sifan Yang, Changshui Zhang, Tianling Ren,
• Abstract summary: Deep networks are prone to catastrophic forgetting during sequential task learning.<n>We propose a novel CL framework, learning without isolation(LwI), where model fusion is formulated as graph matching.<n>Thanks to the sparsity of activation channels in a deep network, LwI can adaptively allocate available pathways for a new task, realizing pathway protection and addressing catastrophic forgetting.
• Score: 64.3476595369537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep networks are prone to catastrophic forgetting during sequential task learning, i.e., losing the knowledge about old tasks upon learning new tasks. To this end, continual learning(CL) has emerged, whose existing methods focus mostly on regulating or protecting the parameters associated with the previous tasks. However, parameter protection is often impractical, since the size of parameters for storing the old-task knowledge increases linearly with the number of tasks, otherwise it is hard to preserve the parameters related to the old-task knowledge. In this work, we bring a dual opinion from neuroscience and physics to CL: in the whole networks, the pathways matter more than the parameters when concerning the knowledge acquired from the old tasks. Following this opinion, we propose a novel CL framework, learning without isolation(LwI), where model fusion is formulated as graph matching and the pathways occupied by the old tasks are protected without being isolated. Thanks to the sparsity of activation channels in a deep network, LwI can adaptively allocate available pathways for a new task, realizing pathway protection and addressing catastrophic forgetting in a parameter-efficient manner. Experiments on popular benchmark datasets demonstrate the superiority of the proposed LwI.

Related papers

• Analytic Subspace Routing: How Recursive Least Squares Works in Continual Learning of Large Language Model [6.42114585934114]
  Large Language Models (LLMs) possess capabilities that can process diverse language-related tasks.<n>Continual Learning in Large Language Models (LLMs) arises which aims to continually adapt the LLMs to new tasks.<n>This paper proposes Analytic Subspace Routing(ASR) to address these challenges.
  arXiv  Detail & Related papers  (2025-03-17T13:40:46Z)
• Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning [22.13331870720021]
  We propose a beyond prompt learning approach to the RFCL task, called Continual Adapter (C-ADA) C-ADA flexibly extends specific weights in CAL to learn new knowledge for each task and freezes old weights to preserve prior knowledge. Our approach achieves significantly improved performance and training speed, outperforming the current state-of-the-art (SOTA) method.
  arXiv  Detail & Related papers  (2024-07-14T17:40:40Z)
• Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning [64.14254712331116]
  Non-exemplar class incremental learning aims to learn both the new and old tasks without accessing any training data from the past. We propose a novel framework of fine-grained knowledge selection and restoration.
  arXiv  Detail & Related papers  (2023-12-20T02:34:11Z)
• CLR: Channel-wise Lightweight Reprogramming for Continual Learning [63.94773340278971]
  Continual learning aims to emulate the human ability to continually accumulate knowledge over sequential tasks. The main challenge is to maintain performance on previously learned tasks after learning new tasks. We propose a Channel-wise Lightweight Reprogramming approach that helps convolutional neural networks overcome catastrophic forgetting.
  arXiv  Detail & Related papers  (2023-07-21T06:56:21Z)
• Context selectivity with dynamic availability enables lifelong continual learning [8.828725115733986]
  The brain is able to learn complex skills, stop the practice for years, learn other skills in between, and still retrieve the original knowledge when necessary. We suggest a bio-plausible meta-plasticity rule building on classical work in lifelong learning. We show that neuron selectivity and neuron-wide consolidation is a simple and viable meta-plasticity hypothesis to enable CL in the brain.
  arXiv  Detail & Related papers  (2023-06-02T17:04:36Z)
• Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning [60.501201259732625]
  We introduce task-adaptive saliency for EFCIL and propose a new framework, which we call Task-Adaptive Saliency Supervision (TASS) Our experiments demonstrate that our method can better preserve saliency maps across tasks and achieve state-of-the-art results on the CIFAR-100, Tiny-ImageNet, and ImageNet-Subset EFCIL benchmarks.
  arXiv  Detail & Related papers  (2022-12-16T02:43:52Z)
• Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer [39.99577526417276]
  In continual learning (CL) an agent can improve the learning performance of both a new task and old' tasks. Most existing CL methods focus on addressing catastrophic forgetting in neural networks by minimizing the modification of the learnt model for old tasks. We propose a new CL method with Backward knowlEdge tRansfer (CUBER) for a fixed capacity neural network without data replay.
  arXiv  Detail & Related papers  (2022-11-01T23:55:51Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• Iterative Network Pruning with Uncertainty Regularization for Lifelong Sentiment Classification [25.13885692629219]
  Lifelong learning is non-trivial for deep neural networks. We propose a novel iterative network pruning with uncertainty regularization method for lifelong sentiment classification.
  arXiv  Detail & Related papers  (2021-06-21T15:34:13Z)
• Continual Learning via Bit-Level Information Preserving [88.32450740325005]
  We study the continual learning process through the lens of information theory. We propose Bit-Level Information Preserving (BLIP) that preserves the information gain on model parameters. BLIP achieves close to zero forgetting while only requiring constant memory overheads throughout continual learning.
  arXiv  Detail & Related papers  (2021-05-10T15:09:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.