Online Continual Learning with Contrastive Vision Transformer
- URL: http://arxiv.org/abs/2207.13516v1
- Date: Sun, 24 Jul 2022 08:51:02 GMT
- Title: Online Continual Learning with Contrastive Vision Transformer
- Authors: Zhen Wang, Liu Liu, Yajing Kong, Jiaxian Guo, and Dacheng Tao
- Abstract summary: This paper proposes a framework Contrastive Vision Transformer (CVT) to achieve a better stability-plasticity trade-off for online CL.
Specifically, we design a new external attention mechanism for online CL that implicitly captures previous tasks' information.
Based on the learnable focuses, we design a focal contrastive loss to rebalance contrastive learning between new and past classes and consolidate previously learned representations.
- Score: 67.72251876181497
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Online continual learning (online CL) studies the problem of learning
sequential tasks from an online data stream without task boundaries, aiming to
adapt to new data while alleviating catastrophic forgetting on the past tasks.
This paper proposes a framework Contrastive Vision Transformer (CVT), which
designs a focal contrastive learning strategy based on a transformer
architecture, to achieve a better stability-plasticity trade-off for online CL.
Specifically, we design a new external attention mechanism for online CL that
implicitly captures previous tasks' information. Besides, CVT contains
learnable focuses for each class, which could accumulate the knowledge of
previous classes to alleviate forgetting. Based on the learnable focuses, we
design a focal contrastive loss to rebalance contrastive learning between new
and past classes and consolidate previously learned representations. Moreover,
CVT contains a dual-classifier structure for decoupling learning current
classes and balancing all observed classes. The extensive experimental results
show that our approach achieves state-of-the-art performance with even fewer
parameters on online CL benchmarks and effectively alleviates the catastrophic
forgetting.
Related papers
- Continual Task Learning through Adaptive Policy Self-Composition [54.95680427960524]
CompoFormer is a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network.
Our experiments reveal that CompoFormer outperforms conventional continual learning (CL) methods, particularly in longer task sequences.
arXiv Detail & Related papers (2024-11-18T08:20:21Z) - What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights [67.72413262980272]
Severe data imbalance naturally exists among web-scale vision-language datasets.
We find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning.
The robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts.
arXiv Detail & Related papers (2024-05-31T17:57:24Z) - Forward-Backward Knowledge Distillation for Continual Clustering [14.234785944941672]
Unsupervised Continual Learning (UCL) is a burgeoning field in machine learning, focusing on enabling neural networks to sequentially learn tasks without explicit label information.
Catastrophic Forgetting (CF) poses a significant challenge in continual learning, especially in UCL, where labeled information of data is not accessible.
We introduce the concept of Unsupervised Continual Clustering (UCC), demonstrating enhanced performance and memory efficiency in clustering across various tasks.
arXiv Detail & Related papers (2024-05-29T16:13:54Z) - Improving Plasticity in Online Continual Learning via Collaborative Learning [22.60291297308379]
We argue that the model's capability to acquire new knowledge (i.e., model plasticity) is another challenge in online CL.
We propose Collaborative Continual Learning (CCL), a collaborative learning based strategy to improve the model's capability in acquiring new concepts.
arXiv Detail & Related papers (2023-12-01T14:06:28Z) - Online Prototype Learning for Online Continual Learning [36.91213307667659]
We study the problem of learning continuously from a single-pass data stream.
By storing a small subset of old data, replay-based methods have shown promising performance.
This paper aims to understand why the online learning models fail to generalize well from a new perspective of shortcut learning.
arXiv Detail & Related papers (2023-08-01T05:46:40Z) - Multi-View Class Incremental Learning [57.14644913531313]
Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance.
This paper investigates a novel paradigm called multi-view class incremental learning (MVCIL), where a single model incrementally classifies new classes from a continual stream of views.
arXiv Detail & Related papers (2023-06-16T08:13:41Z) - On the Effectiveness of Equivariant Regularization for Robust Online
Continual Learning [17.995662644298974]
Continual Learning (CL) approaches seek to bridge this gap by facilitating the transfer of knowledge to both previous tasks and future ones.
Recent research has shown that self-supervision can produce versatile models that can generalize well to diverse downstream tasks.
We propose Continual Learning via Equivariant Regularization (CLER), an OCL approach that leverages equivariant tasks for self-supervision.
arXiv Detail & Related papers (2023-05-05T16:10:31Z) - Mitigating Forgetting in Online Continual Learning via Contrasting
Semantically Distinct Augmentations [22.289830907729705]
Online continual learning (OCL) aims to enable model learning from a non-stationary data stream to continuously acquire new knowledge as well as retain the learnt one.
Main challenge comes from the "catastrophic forgetting" issue -- the inability to well remember the learnt knowledge while learning the new ones.
arXiv Detail & Related papers (2022-11-10T05:29:43Z) - When Does Contrastive Learning Preserve Adversarial Robustness from
Pretraining to Finetuning? [99.4914671654374]
We propose AdvCL, a novel adversarial contrastive pretraining framework.
We show that AdvCL is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency.
arXiv Detail & Related papers (2021-11-01T17:59:43Z) - Incremental Embedding Learning via Zero-Shot Translation [65.94349068508863]
Current state-of-the-art incremental learning methods tackle catastrophic forgetting problem in traditional classification networks.
We propose a novel class-incremental method for embedding network, named as zero-shot translation class-incremental method (ZSTCI)
In addition, ZSTCI can easily be combined with existing regularization-based incremental learning methods to further improve performance of embedding networks.
arXiv Detail & Related papers (2020-12-31T08:21:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.