When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

• URL: http://arxiv.org/abs/2606.28117v1
• Date: Fri, 26 Jun 2026 14:19:03 GMT
• Title: When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
• Abstract summary: We show that task-specific LoRA adapters in Continual Learning exhibit significant low-rank redundancy.<n>We propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations.
• Score: 31.20315349532034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Related papers

• Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning [2.749219725903528]
  Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially.<n>LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately.<n>Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting.<n>Latent-LoRA is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task.
  arXiv  Detail & Related papers  (2026-07-26T20:42:56Z)
• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models [78.56572636138429]
  We propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating.<n>We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
  arXiv  Detail & Related papers  (2026-04-17T15:38:37Z)
• AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition [41.654675205772485]
  We propose a vision-language fine-tuning framework based on cross-layer tensor ring decomposition (TRD) with the integration and collaboration of diverse adapters, called AdaRing.<n>Our experiments show that the proposed AdaRing achieves the state-of-the-art performance while reducing average training parameters by 90%.
  arXiv  Detail & Related papers  (2025-08-16T01:56:27Z)
• Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts [72.22148263683037]
  We study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures.<n>First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature.<n>Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks.
  arXiv  Detail & Related papers  (2025-07-09T03:25:45Z)
• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning [8.81873424028249]
  Class-Incremental Learning (CIL) aims to learn new classes sequentially while retaining the knowledge of previously learned classes.<n>We propose a novel dual-adapter architecture combining textbftask-shared adapters to learn cross-task knowledge and textbftask-specific adapters to capture unique features of each new task.<n>We demonstrate CL-LoRA consistently achieves promising performance under multiple benchmarks with reduced training and inference computation.
  arXiv  Detail & Related papers  (2025-05-30T17:19:52Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• Towards Optimal Adapter Placement for Efficient Transfer Learning [73.1149084352343]
  PETL aims to adapt pre-trained models to new downstream tasks while minimizing the number of fine-tuned parameters. adapters, a popular approach in PETL, inject additional capacity into existing networks by incorporating low-rank projections. This paper investigates the relationship between the placement of an adapter and its performance.
  arXiv  Detail & Related papers  (2024-10-21T10:37:17Z)
• Adaptive Adapter Routing for Long-Tailed Class-Incremental Learning [55.384428765798496]
  New data exhibits a long-tailed distribution, such as e-commerce platform reviews. This necessitates continuous model learning imbalanced data without forgetting. We introduce AdaPtive Adapter RouTing (APART) as an exemplar-free solution for LTCIL.
  arXiv  Detail & Related papers  (2024-09-11T17:52:00Z)
• AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks [55.705355299065474]
  Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.
  arXiv  Detail & Related papers  (2022-04-30T16:49:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.