SPeCiaL: Self-Supervised Pretraining for Continual Learning

• URL: http://arxiv.org/abs/2106.09065v1
• Date: Wed, 16 Jun 2021 18:15:15 GMT
• Title: SPeCiaL: Self-Supervised Pretraining for Continual Learning
• Authors: Lucas Caccia, Joelle Pineau
• Abstract summary: SPeCiaL is a method for unsupervised pretraining of representations tailored for continual learning. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.
• Score: 49.34919926042038
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents SPeCiaL: a method for unsupervised pretraining of representations tailored for continual learning. Our approach devises a meta-learning objective that differentiates through a sequential learning process. Specifically, we train a linear model over the representations to match different augmented views of the same image together, each view presented sequentially. The linear model is then evaluated on both its ability to classify images it just saw, and also on images from previous iterations. This gives rise to representations that favor quick knowledge retention with minimal forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.

Related papers

• Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning [12.5354658533836]
  Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples. For artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge. We propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches.
  arXiv  Detail & Related papers  (2024-05-06T02:02:57Z)
• RanDumb: A Simple Approach that Questions the Efficacy of Continual Representation Learning [68.42776779425978]
  We show that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios.
  arXiv  Detail & Related papers  (2024-02-13T22:07:29Z)
• TExplain: Explaining Learned Visual Features via Pre-trained (Frozen) Language Models [14.019349267520541]
  We propose a novel method that leverages the capabilities of language models to interpret the learned features of pre-trained image classifiers. Our approach generates a vast number of sentences to explain the features learned by the classifier for a given image. Our method, for the first time, utilizes these frequent words corresponding to a visual representation to provide insights into the decision-making process.
  arXiv  Detail & Related papers  (2023-09-01T20:59:46Z)
• SLIP: Self-supervision meets Language-Image Pre-training [79.53764315471543]
  We study whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. We find that SLIP enjoys the best of both worlds: better performance than self-supervision and language supervision.
  arXiv  Detail & Related papers  (2021-12-23T18:07:13Z)
• LiRA: Learning Visual Speech Representations from Audio through Self-supervision [53.18768477520411]
  We propose Learning visual speech Representations from Audio via self-supervision (LiRA) Specifically, we train a ResNet+Conformer model to predict acoustic features from unlabelled visual speech. We show that our approach significantly outperforms other self-supervised methods on the Lip Reading in the Wild dataset.
  arXiv  Detail & Related papers  (2021-06-16T23:20:06Z)
• Class-Balanced Distillation for Long-Tailed Visual Recognition [100.10293372607222]
  Real-world imagery is often characterized by a significant imbalance of the number of images per class, leading to long-tailed distributions. In this work, we introduce a new framework, by making the key observation that a feature representation learned with instance sampling is far from optimal in a long-tailed setting. Our main contribution is a new training method, that leverages knowledge distillation to enhance feature representations.
  arXiv  Detail & Related papers  (2021-04-12T08:21:03Z)
• Distilling Visual Priors from Self-Supervised Learning [24.79633121345066]
  Convolutional Neural Networks (CNNs) are prone to overfit small training datasets. We present a novel two-phase pipeline that leverages self-supervised learning and knowledge distillation to improve the generalization ability of CNN models for image classification under the data-deficient setting.
  arXiv  Detail & Related papers  (2020-08-01T13:07:18Z)
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases [34.02639091680309]
  Recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet.
  arXiv  Detail & Related papers  (2020-07-28T00:11:31Z)
• Distilling Localization for Self-Supervised Representation Learning [82.79808902674282]
  Contrastive learning has revolutionized unsupervised representation learning. Current contrastive models are ineffective at localizing the foreground object. We propose a data-driven approach for learning in variance to backgrounds.
  arXiv  Detail & Related papers  (2020-04-14T16:29:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.