Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning

• URL: http://arxiv.org/abs/2601.21345v1
• Date: Thu, 29 Jan 2026 07:15:05 GMT
• Title: Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning
• Abstract summary: Class-Incremental Learning (CIL) requires a model to continually learn new classes without forgetting old ones.<n>We argue that this parameter-constraining method is detrimental to plasticity.<n>We propose Semantic-Guided Dynamic Sparsification (SGDS), a novel method that proactively guides the activation space.
• Score: 25.808884795490613
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Class-Incremental Learning (CIL) requires a model to continually learn new classes without forgetting old ones. A common and efficient solution freezes a pre-trained model and employs lightweight adapters, whose parameters are often forced to be orthogonal to prevent inter-task interference. However, we argue that this parameter-constraining method is detrimental to plasticity. To this end, we propose Semantic-Guided Dynamic Sparsification (SGDS), a novel method that proactively guides the activation space by governing the orientation and rank of its subspaces through targeted sparsification. Specifically, SGDS promotes knowledge transfer by encouraging similar classes to share a compact activation subspace, while simultaneously preventing interference by assigning non-overlapping activation subspaces to dissimilar classes. By sculpting class-specific sparse subspaces in the activation space, SGDS effectively mitigates interference without imposing rigid constraints on the parameter space. Extensive experiments on various benchmark datasets demonstrate the state-of-the-art performance of SGDS.

Related papers

• Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning [2.749219725903528]
  Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially.<n>LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately.<n>Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting.<n>Latent-LoRA is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task.
  arXiv  Detail & Related papers  (2026-07-26T20:42:56Z)
• Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning [57.237835833636154]
  We propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge.<n>Miles achieves state-of-the-art performance in various CIL settings.
  arXiv  Detail & Related papers  (2026-07-20T06:22:02Z)
• Learning When to Adapt [29.665123668818378]
  Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method.<n>DiseL augments LoRA modules with lightweight input-dependent gates over individual rank-one components.<n>DiseL adds only a small number of parameters and preserves the low-rank structure.
  arXiv  Detail & Related papers  (2026-05-18T18:51:24Z)
• Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models [63.169731309799324]
  Class-incremental learning aims to continuously acquire new knowledge while preserving previously learned information, thereby mitigating catastrophic forgetting.<n>We propose HDSD, a Hierarchical Dual-Subspace Decoupling framework for continual learning in vision-language models.<n> Specifically, we introduce a lightweight Feature Modulation Module (FMM) that explicitly decomposes the parameter space into general and task-specific subspaces.
  arXiv  Detail & Related papers  (2026-05-08T09:42:05Z)
• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models [78.56572636138429]
  We propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating.<n>We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
  arXiv  Detail & Related papers  (2026-04-17T15:38:37Z)
• Shared LoRA Subspaces for almost Strict Continual Learning [32.4267950435704]
  Adapting large pretrained models to new tasks efficiently and continually is crucial for real-world deployment.<n>We propose Share, a novel approach to parameter efficient continual finetuning that learns and dynamically updates a single, shared low-rank subspace.<n>A single Share model can replace hundreds of task-specific LoRA adapters, supporting scalable, asynchronous continual learning.
  arXiv  Detail & Related papers  (2026-02-05T18:59:58Z)
• Action-Constrained Imitation Learning [12.316546911223263]
  Policy learning under action constraints plays a central role in ensuring safe behaviors in various robot control and resource allocation applications.<n>In this paper, we study a new problem setting termed Action-Constrained Imitation Learning (ACIL), where an action-constrained imitator aims to learn from a demonstrative expert with larger action space.<n>We tackle this mismatch through textittrajectory alignment and propose DTWIL, which replaces the original expert demonstrations with a surrogate dataset that follows similar state trajectories while adhering to the action constraints.
  arXiv  Detail & Related papers  (2025-08-20T03:19:07Z)
• Regularizing Subspace Redundancy of Low-Rank Adaptation [54.473090597164834]
  We propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation.<n>Our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks.<n>As a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs.
  arXiv  Detail & Related papers  (2025-07-28T11:52:56Z)
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning [53.88000987041739]
  Class-Incremental Learning (CIL) aims to enable AI models to continuously learn from sequentially arriving data of different classes over time.<n>We propose the Elastic Knowledge Preservation and Compensation (EKPC) method, integrating Importance-aware importance Regularization (IPR) and Trainable Semantic Drift Compensation (TSDC) for CIL.
  arXiv  Detail & Related papers  (2025-06-14T05:19:58Z)
• StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning [79.44594332189018]
  Class-Incremental Learning (CIL) seeks to develop models that continuously learn new action categories over time without previously acquired knowledge.<n>Existing approaches either rely on forgetting, raising concerns over memory and privacy, or adapt static image-based methods that neglect temporal modeling.<n>We propose a unified and exemplar-free VCIL framework that explicitly disentangles and preserves information.
  arXiv  Detail & Related papers  (2025-05-20T06:46:51Z)
• Continuous Knowledge-Preserving Decomposition with Adaptive Layer Selection for Few-Shot Class-Incremental Learning [73.59672160329296]
  CKPD-FSCIL is a unified framework that unlocks the underutilized capacity of pretrained weights.<n>Our method consistently outperforms state-of-the-art approaches in both adaptability and knowledge retention.
  arXiv  Detail & Related papers  (2025-01-09T07:18:48Z)
• Embedding Space Allocation with Angle-Norm Joint Classifiers for Few-Shot Class-Incremental Learning [8.321592316231786]
  Few-shot class-incremental learning aims to continually learn new classes from only a few samples.<n>Current classes occupy the entire feature space, which is detrimental to learning new classes.<n>Small number of samples in incremental rounds is insufficient for fully training.
  arXiv  Detail & Related papers  (2024-11-14T07:31:12Z)
• Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectors [8.761404991620285]
  Activation intervention has emerged as an effective and economical method to modify the behavior of large language models (LLMs)<n>We propose Semantics-Adaptive Dynamic Intervention (SADI), a novel method that constructs a dynamic steering vector to intervene model activations at inference time.<n> Experimental results show that SADI outperforms established baselines by substantial margins, improving task performance without training.
  arXiv  Detail & Related papers  (2024-10-16T06:58:49Z)
• Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning [115.79349923044663]
  Few-shot class-incremental learning (FSCIL) aims to incrementally learn novel classes from limited examples.<n>Existing methods face a critical dilemma: static architectures rely on a fixed parameter space to learn from data that arrive sequentially, prone to overfitting to the current session.<n>In this study, we explore the potential of Selective State Space Models (SSMs) for FSCIL.
  arXiv  Detail & Related papers  (2024-07-08T17:09:39Z)
• Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning [65.57123249246358]
  We propose ExpAndable Subspace Ensemble (EASE) for PTM-based CIL. We train a distinct lightweight adapter module for each new task, aiming to create task-specific subspaces. Our prototype complement strategy synthesizes old classes' new features without using any old class instance.
  arXiv  Detail & Related papers  (2024-03-18T17:58:13Z)
• Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning [60.501201259732625]
  We introduce task-adaptive saliency for EFCIL and propose a new framework, which we call Task-Adaptive Saliency Supervision (TASS) Our experiments demonstrate that our method can better preserve saliency maps across tasks and achieve state-of-the-art results on the CIFAR-100, Tiny-ImageNet, and ImageNet-Subset EFCIL benchmarks.
  arXiv  Detail & Related papers  (2022-12-16T02:43:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.