Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

• URL: http://arxiv.org/abs/2607.23837v1
• Date: Sun, 26 Jul 2026 20:42:56 GMT
• Title: Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
• Abstract summary: Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially.<n>LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately.<n>Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting.<n>Latent-LoRA is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task.
• Score: 2.749219725903528
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Related papers

• One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning [45.405420567098155]
  We present textbfFACET: task-conditioned textbfFetextbfAture transformation with textbfConditiontextbfEd feature consistextbfTency.<n>Our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs.
  arXiv  Detail & Related papers  (2026-08-31T17:03:45Z)
• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• Learning When to Adapt [29.665123668818378]
  Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method.<n>DiseL augments LoRA modules with lightweight input-dependent gates over individual rank-one components.<n>DiseL adds only a small number of parameters and preserves the low-rank structure.
  arXiv  Detail & Related papers  (2026-05-18T18:51:24Z)
• Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA [8.191149161025198]
  Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else.<n>Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision.<n>We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection.
  arXiv  Detail & Related papers  (2026-04-14T09:59:32Z)
• CORAL: Scalable Multi-Task Robot Learning via LoRA Experts [49.759823970016974]
  Vision-Language-Action (VLA) models in real-world robotics expose a core multi-task learning challenge.<n>Maintaining a separate full checkpoint per task is often storage- and deployment-prohibitive.<n>We present CORAL, a backbone- and-agnostic framework designed primarily to mitigate multi-task interference.
  arXiv  Detail & Related papers  (2026-03-10T07:28:41Z)
• The Quest for Winning Tickets in Low-Rank Adapters [24.58659526975649]
  We investigate whether the Lottery Ticket Hypothesis extends to parameter-efficient fine-tuning.<n>Our key finding is that LTH holds within Low-Rank Adaptation (LoRA) methods.<n>We propose Partial-LoRA, a method that identifies saidworks and trains sparse low-rank adapters aligned with task-relevant subspaces of the pre-trained model.
  arXiv  Detail & Related papers  (2025-12-27T06:39:08Z)
• DRL: Discriminative Representation Learning with Parallel Adapters for Class Incremental Learning [63.65467569295623]
  We propose the Discriminative Representation Learning (DRL) framework to specifically address these challenges.<n>To conduct incremental learning effectively and yet efficiently, the DRL's network is built upon a PTM.<n>Our DRL consistently outperforms other state-of-the-art methods throughout the entire CIL period.
  arXiv  Detail & Related papers  (2025-10-14T03:19:15Z)
• CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning [8.81873424028249]
  Class-Incremental Learning (CIL) aims to learn new classes sequentially while retaining the knowledge of previously learned classes.<n>We propose a novel dual-adapter architecture combining textbftask-shared adapters to learn cross-task knowledge and textbftask-specific adapters to capture unique features of each new task.<n>We demonstrate CL-LoRA consistently achieves promising performance under multiple benchmarks with reduced training and inference computation.
  arXiv  Detail & Related papers  (2025-05-30T17:19:52Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Parameter-Efficient Transfer Learning with Diff Pruning [108.03864629388404]
  diff pruning is a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark.
  arXiv  Detail & Related papers  (2020-12-14T12:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.