CLA: Latent Alignment for Online Continual Self-Supervised Learning

• URL: http://arxiv.org/abs/2507.10434v2
• Date: Tue, 15 Jul 2025 09:43:30 GMT
• Title: CLA: Latent Alignment for Online Continual Self-Supervised Learning
• Authors: Giacomo Cignoni, Andrea Cossu, Alexandra Gomez-Villa, Joost van de Weijer, Antonio Carta,
• Abstract summary: We introduce Continual Latent Alignment (CLA), a novel SSL strategy for Online CL.<n>Our CLA is able to speed up the convergence of the training process in the online scenario, outperforming state-of-the-art approaches under the same computational budget.<n>We also discovered that using CLA as a pretraining protocol in the early stages of pretraining leads to a better final performance when compared to a full i.i.d. pretraining.
• Score: 53.52783900926569
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Self-supervised learning (SSL) is able to build latent representations that generalize well to unseen data. However, only a few SSL techniques exist for the online CL setting, where data arrives in small minibatches, the model must comply with a fixed computational budget, and task boundaries are absent. We introduce Continual Latent Alignment (CLA), a novel SSL strategy for Online CL that aligns the representations learned by the current model with past representations to mitigate forgetting. We found that our CLA is able to speed up the convergence of the training process in the online scenario, outperforming state-of-the-art approaches under the same computational budget. Surprisingly, we also discovered that using CLA as a pretraining protocol in the early stages of pretraining leads to a better final performance when compared to a full i.i.d. pretraining.

Related papers

• Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA [19.982853959240497]
  Pre-trained vision-language embedding models such as CLIP have been widely adopted and validated in Continual Learning (CL)<n>Existing CL methods primarily focus on continual downstream adaptation using components isolated from the pre-trained model (PTM)<n>We propose a universal and efficient CL approach for CLIP based on Dynamic Rank-Selective LoRA (CoDyRA)
  arXiv  Detail & Related papers  (2024-12-01T23:41:42Z)
• CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models [23.398619576886375]
  Continual learning (CL) aims to help deep neural networks learn new knowledge while retaining what has been learned. Our work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task.
  arXiv  Detail & Related papers  (2024-03-28T04:15:58Z)
• Read Between the Layers: Leveraging Multi-Layer Representations for Rehearsal-Free Continual Learning with Pre-Trained Models [15.847302755988506]
  We address the Continual Learning problem, wherein a model must learn a sequence of tasks from non-stationary distributions. We propose LayUP, a new prototype-based approach to CL that leverages second-order feature statistics from multiple intermediate layers of a pre-trained network. Our results demonstrate that fully exhausting the representational capacities of pre-trained models in CL goes well beyond their final embeddings.
  arXiv  Detail & Related papers  (2023-12-13T13:11:44Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• On the Effectiveness of Equivariant Regularization for Robust Online Continual Learning [17.995662644298974]
  Continual Learning (CL) approaches seek to bridge this gap by facilitating the transfer of knowledge to both previous tasks and future ones. Recent research has shown that self-supervision can produce versatile models that can generalize well to diverse downstream tasks. We propose Continual Learning via Equivariant Regularization (CLER), an OCL approach that leverages equivariant tasks for self-supervision.
  arXiv  Detail & Related papers  (2023-05-05T16:10:31Z)
• CLIPood: Generalizing CLIP to Out-of-Distributions [73.86353105017076]
  Contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, but the further adaptation of CLIP on downstream tasks undesirably degrades OOD performances. We propose CLIPood, a fine-tuning method that can adapt CLIP models to OOD situations where both domain shifts and open classes may occur on unseen test data. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.
  arXiv  Detail & Related papers  (2023-02-02T04:27:54Z)
• Do Pre-trained Models Benefit Equally in Continual Learning? [25.959813589169176]
  Existing work on continual learning (CL) is primarily devoted to developing algorithms for models trained from scratch. Despite their encouraging performance on contrived benchmarks, these algorithms show dramatic performance drops in real-world scenarios. This paper advocates the systematic introduction of pre-training to CL.
  arXiv  Detail & Related papers  (2022-10-27T18:03:37Z)
• Beyond Supervised Continual Learning: a Review [69.9674326582747]
  Continual Learning (CL) is a flavor of machine learning where the usual assumption of stationary data distribution is relaxed or omitted. Changes in the data distribution can cause the so-called catastrophic forgetting (CF) effect: an abrupt loss of previous knowledge. This article reviews literature that study CL in other settings, such as learning with reduced supervision, fully unsupervised learning, and reinforcement learning.
  arXiv  Detail & Related papers  (2022-08-30T14:44:41Z)
• A Study of Continual Learning Methods for Q-Learning [78.6363825307044]
  We present an empirical study on the use of continual learning (CL) methods in a reinforcement learning (RL) scenario. Our results show that dedicated CL methods can significantly improve learning when compared to the baseline technique of "experience replay"
  arXiv  Detail & Related papers  (2022-06-08T14:51:52Z)
• Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods [61.49061000562676]
  We introduce Cluster Learnability (CL) to assess learnability. CL is measured in terms of the performance of a KNN trained to predict labels obtained by clustering the representations with K-means. We find that CL better correlates with in-distribution model performance than other competing recent evaluation schemes.
  arXiv  Detail & Related papers  (2022-06-02T19:05:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.