Self-Supervised Representation Learning: Introduction, Advances and Challenges

• URL: http://arxiv.org/abs/2110.09327v1
• Date: Mon, 18 Oct 2021 13:51:22 GMT
• Title: Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Authors: Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M. Hospedales
• Abstract summary: Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data.
• Score: 125.38214493654534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of deep learning today. These methods have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pre-training alternatives across a variety of data modalities including image, video, sound, text and graphs. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and compute cost. Finally, we survey the major open challenges in the field that provide fertile ground for future work.

Related papers

• Deep Learning-Based Object Pose Estimation: A Comprehensive Survey [73.74933379151419]
  We discuss the recent advances in deep learning-based object pose estimation. Our survey also covers multiple input data modalities, degrees-of-freedom of output poses, object properties, and downstream tasks.
  arXiv  Detail & Related papers  (2024-05-13T14:44:22Z)
• Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models [7.742594744641462]
  Machine unlearning aims to remove information derived from forgotten data while preserving that of the remaining dataset in a well-trained model. We propose a supervision-free unlearning approach that operates without the need for labels during the unlearning process.
  arXiv  Detail & Related papers  (2024-03-31T00:29:00Z)
• Few-Shot Learning on Graphs: from Meta-learning to Pre-training and Prompting [56.25730255038747]
  This survey endeavors to synthesize recent developments, provide comparative insights, and identify future directions. We systematically categorize existing studies into three major families: meta-learning approaches, pre-training approaches, and hybrid approaches. We analyze the relationships among these methods and compare their strengths and limitations.
  arXiv  Detail & Related papers  (2024-02-02T14:32:42Z)
• Explainable Attention for Few-shot Learning and Beyond [7.044125601403848]
  We introduce a novel framework for achieving explainable hard attention finding, specifically tailored for few-shot learning scenarios. Our approach employs deep reinforcement learning to implement the concept of hard attention, directly impacting raw input data.
  arXiv  Detail & Related papers  (2023-10-11T18:33:17Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• VIPriors 3: Visual Inductive Priors for Data-Efficient Deep Learning Challenges [13.085098213230568]
  Third edition of the "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" workshop featured four data-impaired challenges. Challenges focused on addressing the limitations of data availability in training deep learning models for computer vision tasks.
  arXiv  Detail & Related papers  (2023-05-31T09:31:54Z)
• Self-Supervised Learning of Multi-Object Keypoints for Robotic Manipulation [8.939008609565368]
  In this paper, we demonstrate the efficacy of learning image keypoints via the Dense Correspondence pretext task for downstream policy learning. We evaluate our approach on diverse robot manipulation tasks, compare it to other visual representation learning approaches, and demonstrate its flexibility and effectiveness for sample-efficient policy learning.
  arXiv  Detail & Related papers  (2022-05-17T13:15:07Z)
• Causal Reasoning Meets Visual Representation Learning: A Prospective Study [117.08431221482638]
  Lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models. Inspired by the strong inference ability of human-level agents, recent years have witnessed great effort in developing causal reasoning paradigms. This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods.
  arXiv  Detail & Related papers  (2022-04-26T02:22:28Z)
• Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations [183.03278932562438]
  This paper presents an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We show that our approach achieves higher efficiency in visual representations and thus delivers a key message to inspire the future research of self-supervised visual representation learning.
  arXiv  Detail & Related papers  (2020-11-19T16:26:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.