Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning

• URL: http://arxiv.org/abs/2608.00630v1
• Date: Sat, 01 Aug 2026 12:38:55 GMT
• Title: Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning
• Abstract summary: We introduce a novel measure, the relative parameter-importance, which measures the relative importance of each parameter with respect to both the current and past tasks.<n>Unlike existing methods, our approach allows the update of parameters with high past-task importance when they have low relative-importance.<n>We demonstrate improvements against state-of-the-art CL methods on both class-incremental and domain-incremental learning text classification problems.
• Score: 2.141079906482723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Achieving continual learning (CL) with deep neural networks requires balancing stability and plasticity while enabling knowledge transfer. In this work, we focus on offline learning algorithms under the constraints: (I) no access to training data from prior tasks (II) no access to task-id at inference time. We introduce a novel measure, the relative parameter-importance, which measures the relative importance of each parameter with respect to both the current and past tasks. Parameters with high relative importance are interpreted as more important for maintaining past-task stability and thus heavily regularised, whereas parameters with low relative-importance are allowed to be more freely updated. Unlike existing methods, our approach allows the update of parameters with high past-task importance when they have low relative-importance, thus enabling backward knowledge transfer in addition to tackling the stability-plasticity trade-off. We demonstrate improvements against state-of-the-art CL methods on both class-incremental and domain-incremental learning text classification problems and provide insights for extending our method to text generation problems. Code available at: https://github.com/itsmemala/LACL

Related papers

• Attribution-Guided Continual Learning for Large Language Models [32.34727629126595]
  Large language models (LLMs) often suffer from catastrophic forgetting in continual learning.<n>Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization.<n>We propose an attribution-guided continual fine-tuning framework for LLMs.
  arXiv  Detail & Related papers  (2026-05-06T17:38:13Z)
• Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation [58.3773038915023]
  Continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge.<n>Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge.<n>We address this problem using a projected first-order method compatible with standard deep-dots used in vision-language models.
  arXiv  Detail & Related papers  (2026-01-31T13:27:02Z)
• InTAct: Interval-based Task Activation Consolidation for Continual Learning [8.228364777296195]
  InTAct preserves functional behavior in shared layers without freezing parameters or storing past data.<n>In doing so, InTAct stabilizes the functional role of important neurons rather than directly restricting parameter values.<n>Across diverse domain-incremental benchmarks, InTAct consistently reduces representation drift and improves performance.
  arXiv  Detail & Related papers  (2025-11-21T17:36:12Z)
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning [53.88000987041739]
  Class-Incremental Learning (CIL) aims to enable AI models to continuously learn from sequentially arriving data of different classes over time.<n>We propose the Elastic Knowledge Preservation and Compensation (EKPC) method, integrating Importance-aware importance Regularization (IPR) and Trainable Semantic Drift Compensation (TSDC) for CIL.
  arXiv  Detail & Related papers  (2025-06-14T05:19:58Z)
• Learning without Isolation: Pathway Protection for Continual Learning [64.3476595369537]
  Deep networks are prone to catastrophic forgetting during sequential task learning.<n>We propose a novel CL framework, learning without isolation(LwI), where model fusion is formulated as graph matching.<n>Thanks to the sparsity of activation channels in a deep network, LwI can adaptively allocate available pathways for a new task, realizing pathway protection and addressing catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-05-24T07:16:55Z)
• Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning [19.27175827358111]
  Continual learning in large language models (LLMs) is prone to catastrophic forgetting, where adapting to new tasks significantly degrades performance on previously learned ones.<n>We propose a novel continual full fine-tuning approach leveraging adaptive singular value decomposition (SVD)<n>We evaluate our approach extensively on standard continual learning benchmarks using both encoder-decoder (T5-Large) and decoder-only (LLaMA-2 7B) models.
  arXiv  Detail & Related papers  (2025-04-09T17:59:42Z)
• Dynamic Continual Learning: Harnessing Parameter Uncertainty for Improved Network Adaptation [0.0]
  We propose using parameter-based uncertainty to determine which parameters are relevant to a network's learned function.<n>We show improved Continual Learning performance for Average Test Accuracy and Backward Transfer metrics.
  arXiv  Detail & Related papers  (2025-01-18T19:58:53Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Continual Learning via Bit-Level Information Preserving [88.32450740325005]
  We study the continual learning process through the lens of information theory. We propose Bit-Level Information Preserving (BLIP) that preserves the information gain on model parameters. BLIP achieves close to zero forgetting while only requiring constant memory overheads throughout continual learning.
  arXiv  Detail & Related papers  (2021-05-10T15:09:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.