Is Parameter Isolation Better for Prompt-Based Continual Learning?

• URL: http://arxiv.org/abs/2601.20894v1
• Date: Wed, 28 Jan 2026 08:17:11 GMT
• Title: Is Parameter Isolation Better for Prompt-Based Continual Learning?
• Authors: Jiangyang Li, Chenhao Ding, Songlin Dong, Qiang Wang, Jianchao Zhao, Yuhang He, Yihong Gong,
• Abstract summary: Most existing methods assign a fixed set of prompts to each task, isolating knowledge across tasks and resulting in suboptimal parameter utilization.<n>This framework constructs a global prompt pool and introduces a task-aware gated routing mechanism that sparsely activates a subset of prompts.<n>We also introduce a history-aware modulator that leverages cumulative prompt activation statistics to protect frequently used prompts from excessive updates.
• Score: 46.254917907419895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt-based continual learning methods effectively mitigate catastrophic forgetting. However, most existing methods assign a fixed set of prompts to each task, completely isolating knowledge across tasks and resulting in suboptimal parameter utilization. To address this, we consider the practical needs of continual learning and propose a prompt-sharing framework. This framework constructs a global prompt pool and introduces a task-aware gated routing mechanism that sparsely activates a subset of prompts to achieve dynamic decoupling and collaborative optimization of task-specific feature representations. Furthermore, we introduce a history-aware modulator that leverages cumulative prompt activation statistics to protect frequently used prompts from excessive updates, thereby mitigating inefficient parameter usage and knowledge forgetting. Extensive analysis and empirical results demonstrate that our approach consistently outperforms existing static allocation strategies in effectiveness and efficiency.

Related papers

• A Unified Multi-Task Learning Framework for Generative Auto-Bidding with Validation-Aligned Optimization [51.27959658504722]
  Multi-task learning offers a principled framework to train these tasks jointly through shared representations.<n>Existing multi-task optimization strategies are primarily guided by training dynamics and often generalize poorly in volatile bidding environments.<n>We present Validation-Aligned Multi-task Optimization (VAMO), which adaptively assigns task weights based on the alignment between per-task training gradients and a held-out validation gradient.
  arXiv  Detail & Related papers  (2025-10-09T03:59:51Z)
• One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning [52.966712416640085]
  We propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies.<n>SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-29T08:54:58Z)
• Dynamic Prompt Fusion for Multi-Task and Cross-Domain Adaptation in LLMs [2.852258765983155]
  This study introduces a unified multi-task learning framework with dynamic prompt scheduling mechanism.<n>It enhances the model's ability to capture semantic differences across tasks.<n>It incorporates an automatic learning strategy for scheduling weights, which effectively mitigates task interference and negative transfer.
  arXiv  Detail & Related papers  (2025-09-09T23:42:16Z)
• Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective [65.12150411762273]
  We show that pruning random demonstrations into seemingly incoherent "gibberish" can remarkably improve performance across diverse tasks.<n>We propose a self-discover prompt optimization framework, PromptQuine, that automatically searches for the pruning strategy by itself using only low-data regimes.
  arXiv  Detail & Related papers  (2025-06-22T07:53:07Z)
• Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learning [76.32953653161417]
  Class-incremental learning enables models to learn new classes progressively while preserving knowledge of previously learned ones.<n>Recent advances in this field have shifted towards parameter-efficient fine-tuning techniques.<n>We present a novel prompt-based approach that addresses the limitation of current approaches.
  arXiv  Detail & Related papers  (2025-03-11T02:27:37Z)
• A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models [14.483240353801074]
  This paper proposes an optimal learning framework for automated prompt engineering.<n>It is designed to sequentially identify effective prompt features while efficiently allocating a limited evaluation budget.<n>Our framework provides a solution to deploying automated prompt engineering in a wider range applications.
  arXiv  Detail & Related papers  (2025-01-07T03:51:10Z)
• Adaptive Prompting for Continual Relation Extraction: A Within-Task Variance Perspective [23.79259400522239]
  We propose a novel approach to address catastrophic forgetting in Continual Relation Extraction.<n>Our approach employs a prompt pool for each task, capturing variations within each task while enhancing cross-task variances.
  arXiv  Detail & Related papers  (2024-12-11T11:00:33Z)
• PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer [76.39111896665585]
  Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
  arXiv  Detail & Related papers  (2024-07-04T10:37:58Z)
• Streaming LifeLong Learning With Any-Time Inference [36.3326483579511]
  We propose a novel lifelong learning approach, which is streaming, i.e., a single input sample arrives in each time step, single pass, class-incremental, and subject to be evaluated at any moment. We additionally propose an implicit regularizer in the form of snap-shot self-distillation, which effectively minimizes the forgetting further. Our empirical evaluations and ablations demonstrate that the proposed method outperforms the prior works by large margins.
  arXiv  Detail & Related papers  (2023-01-27T18:09:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.