Rethinking the Representational Continuity: Towards Unsupervised Continual Learning

• URL: http://arxiv.org/abs/2110.06976v1
• Date: Wed, 13 Oct 2021 18:38:06 GMT
• Title: Rethinking the Representational Continuity: Towards Unsupervised Continual Learning
• Authors: Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, Sung Ju Hwang
• Abstract summary: Unsupervised continual learning (UCL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. We show that reliance on annotated data is not necessary for continual learning. We propose Lifelong Unsupervised Mixup (LUMP) to alleviate catastrophic forgetting for unsupervised representations.
• Score: 45.440192267157094
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent advances in continual learning are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (LUMP), a simple yet effective technique that leverages the interpolation between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations.

Related papers

• Zero-Shot Generalization during Instruction Tuning: Insights from Similarity and Granularity [84.12126298229866]
  We show that zero-shot generalization during instruction tuning happens very early. We also show that encountering highly similar and fine-grained training data earlier during instruction tuning, without the constraints of defined "tasks", enables better generalization. For the first time, we show that zero-shot generalization during instruction tuning is a form of similarity-based generalization between training and test data at the instance level.
  arXiv  Detail & Related papers  (2024-06-17T16:40:21Z)
• DELTA: Decoupling Long-Tailed Online Continual Learning [7.507868991415516]
  Long-Tailed Online Continual Learning (LTOCL) aims to learn new tasks from sequentially arriving class-imbalanced data streams. We present DELTA, a decoupled learning approach designed to enhance learning representations. We demonstrate that DELTA improves the capacity for incremental learning, surpassing existing OCL methods.
  arXiv  Detail & Related papers  (2024-04-06T02:33:04Z)
• Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations [1.8130068086063336]
  We contribute to the field of unsupervised (visual) representation learning from three perspectives. We design unsupervised, backpropagation-free Convolutional Self-Organizing Neural Networks (CSNNs) We build upon the widely used (non-)linear evaluation protocol to define pretext- and target-objective-independent metrics. We contribute CARLANE, the first 3-way sim-to-real domain adaptation benchmark for 2D lane detection, and a method based on self-supervised learning.
  arXiv  Detail & Related papers  (2023-11-30T15:57:55Z)
• Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality [55.88910947643436]
  Self-supervised pre-training is essential for handling vast quantities of unlabeled data in practice. HiDe-Prompt is an innovative approach that explicitly optimize the hierarchical components with an ensemble of task-specific prompts and statistics. Our experiments demonstrate the superior performance of HiDe-Prompt and its robustness to pre-training paradigms in continual learning.
  arXiv  Detail & Related papers  (2023-10-11T06:51:46Z)
• Augment to Interpret: Unsupervised and Inherently Interpretable Graph Embeddings [0.0]
  In this paper, we study graph representation learning and we show that data augmentation that preserves semantics can be learned and used to produce interpretations. Our framework, which we named INGENIOUS, creates inherently interpretable embeddings and eliminates the need for costly additional post-hoc analysis.
  arXiv  Detail & Related papers  (2023-09-28T16:21:40Z)
• Unsupervised Representation Learning for Time Series: A Review [20.00853543048447]
  Unsupervised representation learning approaches aim to learn discriminative feature representations from unlabeled data, without the requirement of annotating every sample. We conduct a literature review of existing rapidly evolving unsupervised representation learning approaches for time series. We empirically evaluate state-of-the-art approaches, especially the rapidly evolving contrastive learning methods, on 9 diverse real-world datasets.
  arXiv  Detail & Related papers  (2023-08-03T07:28:06Z)
• Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approach [14.674494335647841]
  We propose a two-stage training paradigm for CL that intertwines task-agnostic and task-specific learning. We show that our training paradigm can be easily added to memory- or regularization-based approaches.
  arXiv  Detail & Related papers  (2022-07-13T15:16:51Z)
• Feature Forgetting in Continual Representation Learning [48.89340526235304]
  representations do not suffer from "catastrophic forgetting" even in plain continual learning, but little further fact is known about its characteristics. We devise a protocol for evaluating representation in continual learning, and then use it to present an overview of the basic trends of continual representation learning. To study the feature forgetting problem, we create a synthetic dataset to identify and visualize the prevalence of feature forgetting in neural networks.
  arXiv  Detail & Related papers  (2022-05-26T13:38:56Z)
• Co$^2$L: Contrastive Continual Learning [69.46643497220586]
  Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks. We propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations.
  arXiv  Detail & Related papers  (2021-06-28T06:14:38Z)
• Progressive Stage-wise Learning for Unsupervised Feature Representation Enhancement [83.49553735348577]
  We propose the Progressive Stage-wise Learning (PSL) framework for unsupervised learning. Our experiments show that PSL consistently improves results for the leading unsupervised learning methods.
  arXiv  Detail & Related papers  (2021-06-10T07:33:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.