Attribution-Guided Continual Learning for Large Language Models

• URL: http://arxiv.org/abs/2605.05285v1
• Date: Wed, 06 May 2026 17:38:13 GMT
• Title: Attribution-Guided Continual Learning for Large Language Models
• Abstract summary: Large language models (LLMs) often suffer from catastrophic forgetting in continual learning.<n>Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization.<n>We propose an attribution-guided continual fine-tuning framework for LLMs.
• Score: 32.34727629126595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack semantic awareness of internal knowledge distribution in LLMs. As a result, they cannot distinguish parameters that should be preserved or updated. We propose an attribution-guided continual fine-tuning framework for LLMs. Our method estimates task-specific, element-wise parameter importance in each Transformer layer and uses these scores to modulate gradients. Parameters important to previous tasks receive smaller updates, while less relevant ones remain plastic for learning new tasks. Experiments on continual learning benchmarks show that our method consistently outperforms baselines, achieving better retention of old tasks while maintaining competitive performance on new tasks.

Related papers

• TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning [33.07713704586859]
  We introduce TRACE, a novel approach for discovering Task-specific paRameters.<n>We conduct experiments across multiple standard benchmarks to demonstrate the superior performance of our proposed method.
  arXiv  Detail & Related papers  (2026-05-29T08:57:06Z)
• Learning, Fast and Slow: Towards LLMs That Adapt Continually [83.9214051113102]
  Large language models (LLMs) are trained for downstream tasks by updating their parameters.<n>Fast-slow training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-12T17:58:20Z)
• Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation [58.3773038915023]
  Continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge.<n>Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge.<n>We address this problem using a projected first-order method compatible with standard deep-dots used in vision-language models.
  arXiv  Detail & Related papers  (2026-01-31T13:27:02Z)
• Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings [13.325021114990241]
  Catastrophic forgetting remains a critical challenge in continual learning for large language models.<n>We propose Dynamic Orthogonal Continual (DOC) fine-tuning, a novel approach that tracks the drift of functional directions and dynamically updates them during the fine-tuning process.
  arXiv  Detail & Related papers  (2025-09-28T13:55:05Z)
• Continuous Subspace Optimization for Continual Learning [24.597922531045846]
  Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge.<n>We propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the model in a series of subspaces rather than a single one.<n>CoSO significantly outperforms state-of-the-art methods, especially in challenging scenarios with long task sequences.
  arXiv  Detail & Related papers  (2025-05-17T03:53:21Z)
• Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning [19.27175827358111]
  Continual learning in large language models (LLMs) is prone to catastrophic forgetting, where adapting to new tasks significantly degrades performance on previously learned ones.<n>We propose a novel continual full fine-tuning approach leveraging adaptive singular value decomposition (SVD)<n>We evaluate our approach extensively on standard continual learning benchmarks using both encoder-decoder (T5-Large) and decoder-only (LLaMA-2 7B) models.
  arXiv  Detail & Related papers  (2025-04-09T17:59:42Z)
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging [80.17238673443127]
  LiNeS is a post-training editing technique designed to preserve pre-trained generalization while enhancing fine-tuned task performance.<n>LiNeS demonstrates significant improvements in both single-task and multi-task settings across various benchmarks in vision and natural language processing.
  arXiv  Detail & Related papers  (2024-10-22T16:26:05Z)
• Orthogonal Subspace Learning for Language Model Continual Learning [45.35861158925975]
  O-LoRA is a simple and efficient approach for continual learning in language models. Our method induces only marginal additional parameter costs and requires no user data storage for replay.
  arXiv  Detail & Related papers  (2023-10-22T02:23:44Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• CLR: Channel-wise Lightweight Reprogramming for Continual Learning [63.94773340278971]
  Continual learning aims to emulate the human ability to continually accumulate knowledge over sequential tasks. The main challenge is to maintain performance on previously learned tasks after learning new tasks. We propose a Channel-wise Lightweight Reprogramming approach that helps convolutional neural networks overcome catastrophic forgetting.
  arXiv  Detail & Related papers  (2023-07-21T06:56:21Z)
• TAG: Task-based Accumulated Gradients for Lifelong learning [21.779858050277475]
  We propose a task-aware system that adapts the learning rate based on the relatedness among tasks. We empirically show that our proposed adaptive learning rate not only accounts for catastrophic forgetting but also allows positive backward transfer.
  arXiv  Detail & Related papers  (2021-05-11T16:10:32Z)
• Continual Learning via Bit-Level Information Preserving [88.32450740325005]
  We study the continual learning process through the lens of information theory. We propose Bit-Level Information Preserving (BLIP) that preserves the information gain on model parameters. BLIP achieves close to zero forgetting while only requiring constant memory overheads throughout continual learning.
  arXiv  Detail & Related papers  (2021-05-10T15:09:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.