SR$^2$-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning

• URL: http://arxiv.org/abs/2605.07420v1
• Date: Fri, 08 May 2026 08:14:50 GMT
• Title: SR$^2$-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning
• Abstract summary: We present a novel perspective on catastrophic forgetting through the analysis of inter-layer relation drift.<n>We propose underlineSelf-underlineRectifying inter-layer underlineRelation Low-Rank Adaptation(SR$2$-LoRA)<n>Experiments on standard CIL benchmarks demonstrate that SR$2$-LoRA effectively mitigates catastrophic forgetting.
• Score: 3.9562791462574602
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained models with parameter-efficient fine-tuning (PEFT) have demonstrated promising potential for class-incremental learning (CIL), yet catastrophic forgetting still persists when adapting models to new tasks. In this paper, we present a novel perspective on catastrophic forgetting through the analysis of inter-layer relation drift, i.e., the progressive disruption of relationships among layer-wise representations during the learning of new tasks. We theoretically show that the increase of such drift reduces the classification margins of previously learned tasks, thereby degrading overall model performance. To address this issue, we propose \underline{S}elf-\underline{R}ectifying inter-layer \underline{R}elation Low-Rank Adaptation~(SR$^2$-LoRA), a simple yet effective method that mitigates catastrophic forgetting by constraining inter-layer relation drift. Specifically, SR$^2$-LoRA constructs the relation matrices induced by the previous and current models on current-task samples, and aligns the corresponding singular values. We further theoretically show that this alignment exhibits greater robustness to estimation perturbations than direct entry-wise alignment. Extensive experiments on standard CIL benchmarks demonstrate that SR$^2$-LoRA effectively mitigates catastrophic forgetting, with its advantages becoming more pronounced as the number of tasks increases. Code is available in the \href{https://github.com/FqWan24/SR-2-LoRA}{repository}.

Related papers

• Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning [122.06915301391616]
  Low-Rank Adaptation (LoRA) has emerged as a promising paradigm for Continual Learning.<n>We propose textbfJanus-LoRA, a framework that restores this balance through two novel components.
  arXiv  Detail & Related papers  (2026-05-27T13:53:14Z)
• Convergent Stochastic Training of Attention and Understanding LoRA [40.81245220414135]
  Transformers have revolutionized machine learning and deploying attention layers in the model is increasingly standard across a myriad of applications.<n>For large models, it is common to implement Low Rank Adaptation (LoRA), whereby a factorized parameterization of them is trained, to achieve a surprisingly beneficial accuracy-size tradeoff.<n>We prove that for any mild regularization, the empirical regression loss on a attention layer and LoRA on a shallow neural net induce Poincaré inequality for the corresponding Gibbs' measure.
  arXiv  Detail & Related papers  (2026-05-08T16:22:08Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA [50.97792275353563]
  We introduce a novel framework that restructures a single Low-Rank Adaptation (LoRA) module as a decomposable Rank-1 Expert Pool.<n>Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [Guided] token.
  arXiv  Detail & Related papers  (2026-01-30T10:54:51Z)
• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning [57.514786046966265]
  We propose textbfPerturb-and-Merge (P&M), a novel continual learning framework that integrates model merging into the CL paradigm to mitigate forgetting.<n>Our proposed approach achieves state-of-the-art performance on several continual learning benchmark datasets.
  arXiv  Detail & Related papers  (2025-05-28T14:14:19Z)
• SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization [2.594346658179846]
  Low-Rank Adaptation (LoRA) constrains updates to a fixed low-rank subspace.<n>We introduce Subspace Recomposition in Low-Rank Adaptation (SRLoRA) via importance-based fusion and reinitialization.<n> SRLoRA consistently achieves faster convergence and improved accuracy over standard LoRA.
  arXiv  Detail & Related papers  (2025-05-18T14:12:40Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees [63.18324983384337]
  We introduce the first learning to rank method for Gradient Boosted Decision Trees (GBDTs) Our main contribution is a novel estimator for the second-order derivatives, i.e., the Hessian matrix. We incorporate our estimator into the existing PL-Rank framework, which was originally designed for first-order derivatives only.
  arXiv  Detail & Related papers  (2024-04-18T13:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.