KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation

• URL: http://arxiv.org/abs/2606.16256v1
• Date: Mon, 15 Jun 2026 05:57:59 GMT
• Title: KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation
• Abstract summary: Continual learning for pre-trained vision-language models requires balancing three competing objectives.<n>This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism.
• Score: 67.23448686598414
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism. We analyze knowledge distribution of Transformer architecture from both inter-layer and intra-layer perspectives. The inter-layer perspective examines how retention is distributed across layers, while the intra-layer perspective focuses on the parameter space within each layer. Our analysis reveals a structural property: general transferable knowledge is mainly encoded in the shallow layers and the principal subspace of the parameters, while task-specific adaptations are localized in the deep layers and the residual subspace. Motivated by this insight, KeepLoRA++ introduces a layer-scaled residual gradient adaptation method. New tasks are learned by restricting LoRA parameter updates to the residual subspace, combined with a shallow-to-deep layer scaling, to prevent interference with previously acquired capabilities. Specifically, the gradient of a new task is projected onto a subspace orthogonal to both the principal subspace of the pre-trained model and the dominant directions of previous task features, while simultaneously assigning smaller update magnitudes to shallow layers and larger ones to deeper layers. Our theoretical analysis and empirical evaluations confirm that KeepLoRA++ successfully balances these three competing objectives, consistently outperforming representative baselines across image classification, visual question answering, and video understanding tasks.

Related papers

• Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models [63.169731309799324]
  Class-incremental learning aims to continuously acquire new knowledge while preserving previously learned information, thereby mitigating catastrophic forgetting.<n>We propose HDSD, a Hierarchical Dual-Subspace Decoupling framework for continual learning in vision-language models.<n> Specifically, we introduce a lightweight Feature Modulation Module (FMM) that explicitly decomposes the parameter space into general and task-specific subspaces.
  arXiv  Detail & Related papers  (2026-05-08T09:42:05Z)
• DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles [8.564506908883667]
  We propose textbfDeAR, a framework that achieves fine-grained VLM adaptation by textbfDecomposing textbfAttention head textbfRoles.<n>We show that DeAR achieves a strong balance between task adaptation and generalization, outperforming previous methods across various tasks.
  arXiv  Detail & Related papers  (2026-03-01T13:52:51Z)
• Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning [82.30237756328596]
  Low-Rank Adaptation (LoRA) has gained increasing attention in Continual Learning (CL)<n>Several LoRA-based CL methods reduce interference across tasks by separating their update spaces.<n>LoDA performs a task-driven decomposition to build general and truly task-specific LoRA subspaces.
  arXiv  Detail & Related papers  (2026-02-27T02:31:00Z)
• KeepLoRA: Continual Learning with Residual Gradient Adaptation [70.16296045857659]
  Continual learning for pre-trained vision-language models requires balancing three competing objectives.<n>This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives.
  arXiv  Detail & Related papers  (2026-01-27T14:38:57Z)
• Beyond the final layer: Attentive multilayer fusion for vision transformers [45.627646781613386]
  We show that task-relevant information is distributed across the network hierarchy rather than solely encoded in any of the last layers.<n>We apply an attentive probing mechanism that dynamically fuses representations from all layers of a Vision Transformer.<n>This mechanism learns to identify the most relevant layers for a target task and combines low-level structural cues with high-level semantic abstractions.
  arXiv  Detail & Related papers  (2026-01-14T09:50:09Z)
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging [80.17238673443127]
  LiNeS is a post-training editing technique designed to preserve pre-trained generalization while enhancing fine-tuned task performance.<n>LiNeS demonstrates significant improvements in both single-task and multi-task settings across various benchmarks in vision and natural language processing.
  arXiv  Detail & Related papers  (2024-10-22T16:26:05Z)
• An Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecture [0.0]
  This work presents a two-stage adaptive framework for developing deep neural network (DNN) architectures that generalize well for a given training data set. In the first stage, a layerwise training approach is adopted where a new layer is added each time and trained independently by freezing parameters in the previous layers. We introduce a epsilon-delta stability-promoting concept as a desirable property for a learning algorithm and show that employing manifold regularization yields a epsilon-delta stability-promoting algorithm.
  arXiv  Detail & Related papers  (2022-11-13T09:51:16Z)
• Gradient Projection Memory for Continual Learning [5.43185002439223]
  The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. We propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks.
  arXiv  Detail & Related papers  (2021-03-17T16:31:29Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• LSM: Learning Subspace Minimization for Low-level Vision [78.27774638569218]
  We replace the regularization term with a learnable subspace constraint, and preserve the data term to exploit domain knowledge. This learning subspace minimization (LSM) framework unifies the network structures and the parameters for many low-level vision tasks. We demonstrate our LSM framework on four low-level tasks including interactive image segmentation, video segmentation, stereo matching, and optical flow, and validate the network on various datasets.
  arXiv  Detail & Related papers  (2020-04-20T10:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.