RanDumb: A Simple Approach that Questions the Efficacy of Continual Representation Learning

• URL: http://arxiv.org/abs/2402.08823v2
• Date: Tue, 23 Jul 2024 13:52:28 GMT
• Title: RanDumb: A Simple Approach that Questions the Efficacy of Continual Representation Learning
• Authors: Ameya Prabhu, Shiven Sinha, Ponnurangam Kumaraguru, Philip H. S. Torr, Ozan Sener, Puneet K. Dokania,
• Abstract summary: We show that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios.
• Score: 68.42776779425978
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the efficacy of continually learned representations, as representations are learned alongside classifiers throughout the learning process. Our primary contribution is empirically demonstrating that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. Our approach embeds raw pixels using a fixed random transform, approximating an RBF-Kernel initialized before any data is seen. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. This method, called RanDumb, significantly outperforms state-of-the-art continually learned representations across all standard online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios. Extending our investigation to popular exemplar-free scenarios with pretrained models, we find that training only a linear classifier on top of pretrained representations surpasses most continual fine-tuning and prompt-tuning strategies. Overall, our investigation challenges the prevailing assumptions about effective representation learning in online continual learning. Our code is available at://github.com/drimpossible/RanDumb.

Related papers

• Learning Equi-angular Representations for Online Continual Learning [28.047867978274358]
  In particular, we induce neural collapse to form a simplex equiangular tight frame (ETF) structure in the representation space. We show that our proposed method outperforms state-of-the-art methods by a noticeable margin in various online continual learning scenarios.
  arXiv  Detail & Related papers  (2024-04-02T04:29:01Z)
• A Study of Forward-Forward Algorithm for Self-Supervised Learning [65.268245109828]
  We study the performance of forward-forward vs. backpropagation for self-supervised representation learning. Our main finding is that while the forward-forward algorithm performs comparably to backpropagation during (self-supervised) training, the transfer performance is significantly lagging behind in all the studied settings.
  arXiv  Detail & Related papers  (2023-09-21T10:14:53Z)
• A Comprehensive Empirical Evaluation on Online Continual Learning [20.39495058720296]
  We evaluate methods from the literature that tackle online continual learning. We focus on the class-incremental setting in the context of image classification. We compare these methods on the Split-CIFAR100 and Split-TinyImagenet benchmarks.
  arXiv  Detail & Related papers  (2023-08-20T17:52:02Z)
• DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning [29.80680408934347]
  We propose an alternative framework to incremental learning where we continually fine-tune the model from a pre-trained representation. Our method takes advantage of linearization technique of a pre-trained neural network for simple and effective continual learning. We show that our method can be applied to general continual learning settings, we evaluate our method in data-incremental, task-incremental, and class-incremental learning problems.
  arXiv  Detail & Related papers  (2022-08-17T06:58:14Z)
• Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism [65.46524775457928]
  offline reinforcement learning seeks to utilize offline/historical data to optimize sequential decision-making strategies. We study the statistical limits of offline reinforcement learning with linear model representations.
  arXiv  Detail & Related papers  (2022-03-11T09:00:12Z)
• Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data [101.6195176510611]
  "Online" continual learning enables evaluating both information retention and online learning efficacy. In online continual learning, each incoming small batch of data is first used for testing and then added to the training set, making the problem truly online. We introduce a new benchmark for online continual visual learning that exhibits large scale and natural distribution shifts.
  arXiv  Detail & Related papers  (2021-08-20T06:17:20Z)
• Reducing Representation Drift in Online Continual Learning [87.71558506591937]
  We study the online continual learning paradigm, where agents must learn from a changing distribution with constrained memory and compute. In this work we instead focus on the change in representations of previously observed data due to the introduction of previously unobserved class samples in the incoming data stream.
  arXiv  Detail & Related papers  (2021-04-11T15:19:30Z)
• Self-Supervised Training Enhances Online Continual Learning [37.91734641808391]
  In continual learning, a system must incrementally learn from a non-stationary data stream without catastrophic forgetting. Self-supervised pre-training could yield features that generalize better than supervised learning. Our best system achieves a 14.95% relative increase in top-1 accuracy on class incremental ImageNet over the prior state of the art for online continual learning.
  arXiv  Detail & Related papers  (2021-03-25T17:45:27Z)
• Fully Convolutional Networks for Continuous Sign Language Recognition [83.85895472824221]
  Continuous sign language recognition is a challenging task that requires learning on both spatial and temporal dimensions. We propose a fully convolutional network (FCN) for online SLR to concurrently learn spatial and temporal features from weakly annotated video sequences.
  arXiv  Detail & Related papers  (2020-07-24T08:16:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.