Compensating Distribution Drifts in Class-incremental Learning of Pre-trained Vision Transformers

• URL: http://arxiv.org/abs/2511.09926v1
• Date: Fri, 14 Nov 2025 01:19:06 GMT
• Title: Compensating Distribution Drifts in Class-incremental Learning of Pre-trained Vision Transformers
• Authors: Xuan Rao, Simian Xu, Zheng Li, Bo Zhao, Derong Liu, Mingming Ha, Cesare Alippi,
• Abstract summary: We introduce a latent space transition operator and propose Sequential Learning with Drift Compensation.<n>SLDC aims to align feature distributions across tasks to mitigate the impact of drift.<n>Experiments on standard CIL benchmarks demonstrate that SLDC significantly improves the performance of SeqFT.
• Score: 27.14203097630326
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances have shown that sequential fine-tuning (SeqFT) of pre-trained vision transformers (ViTs), followed by classifier refinement using approximate distributions of class features, can be an effective strategy for class-incremental learning (CIL). However, this approach is susceptible to distribution drift, caused by the sequential optimization of shared backbone parameters. This results in a mismatch between the distributions of the previously learned classes and that of the updater model, ultimately degrading the effectiveness of classifier performance over time. To address this issue, we introduce a latent space transition operator and propose Sequential Learning with Drift Compensation (SLDC). SLDC aims to align feature distributions across tasks to mitigate the impact of drift. First, we present a linear variant of SLDC, which learns a linear operator by solving a regularized least-squares problem that maps features before and after fine-tuning. Next, we extend this with a weakly nonlinear SLDC variant, which assumes that the ideal transition operator lies between purely linear and fully nonlinear transformations. This is implemented using learnable, weakly nonlinear mappings that balance flexibility and generalization. To further reduce representation drift, we apply knowledge distillation (KD) in both algorithmic variants. Extensive experiments on standard CIL benchmarks demonstrate that SLDC significantly improves the performance of SeqFT. Notably, by combining KD to address representation drift with SLDC to compensate distribution drift, SeqFT achieves performance comparable to joint training across all evaluated datasets. Code: https://github.com/raoxuan98-hash/sldc.git.

Related papers

• DRL: Discriminative Representation Learning with Parallel Adapters for Class Incremental Learning [63.65467569295623]
  We propose the Discriminative Representation Learning (DRL) framework to specifically address these challenges.<n>To conduct incremental learning effectively and yet efficiently, the DRL's network is built upon a PTM.<n>Our DRL consistently outperforms other state-of-the-art methods throughout the entire CIL period.
  arXiv  Detail & Related papers  (2025-10-14T03:19:15Z)
• No Alignment Needed for Generation: Learning Linearly Separable Representations in Diffusion Models [4.511561231517167]
  We propose an alternative regularization for training, based on promoting the Linear SEParability (LSEP) of intermediate layer representations.<n>Our results demonstrate substantial improvements in both training efficiency and generation quality on flow-based transformer architectures.
  arXiv  Detail & Related papers  (2025-09-25T20:46:48Z)
• A Trainable Optimizer [18.195022468462753]
  We present a framework that jointly trains the full gradient estimator and the trainable weights of the model.<n>Pseudo-linear TO incurs negligible computational overhead, requiring only minimal additional multiplications.<n> Experiments demonstrate that TO methods converge faster than benchmark algorithms.
  arXiv  Detail & Related papers  (2025-08-03T14:06:07Z)
• Exemplar-free Continual Representation Learning via Learnable Drift Compensation [24.114984920918715]
  We propose Learnable Drift Compensation (LDC), which can effectively mitigate drift in any moving backbone. LDC is fast and straightforward to integrate on top of existing continual learning approaches. We achieve state-of-the-art performance in both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2024-07-11T14:23:08Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration [38.4461170690033]
  We propose a novel fine-tuning framework, namely distribution regularization with semantic calibration (DR-Tune) DR-Tune employs distribution regularization by enforcing the downstream task head to decrease its classification error on the pretrained feature distribution. To alleviate the interference by semantic drift, we develop the semantic calibration (SC) module.
  arXiv  Detail & Related papers  (2023-08-23T10:59:20Z)
• Kalman Filter for Online Classification of Non-Stationary Data [101.26838049872651]
  In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps. We introduce a probabilistic Bayesian online learning model by using a neural representation and a state space model over the linear predictor weights. In experiments in multi-class classification we demonstrate the predictive ability of the model and its flexibility to capture non-stationarity.
  arXiv  Detail & Related papers  (2023-06-14T11:41:42Z)
• LQF: Linear Quadratic Fine-Tuning [114.3840147070712]
  We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification.
  arXiv  Detail & Related papers  (2020-12-21T06:40:20Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.