Weak Augmentation Guided Relational Self-Supervised Learning

• URL: http://arxiv.org/abs/2203.08717v3
• Date: Mon, 3 Jun 2024 12:06:06 GMT
• Title: Weak Augmentation Guided Relational Self-Supervised Learning
• Authors: Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, Chang Xu,
• Abstract summary: We introduce a novel relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Our proposed method employs sharpened distribution of pairwise similarities among different instances as textitrelation metric. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures.
• Score: 80.0680103295137
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most methods mainly focus on the instance level information (\ie, the different augmented images of the same instance should have the same feature or cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduce a novel SSL paradigm, which we term as relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, our proposed method employs sharpened distribution of pairwise similarities among different instances as \textit{relation} metric, which is thus utilized to match the feature embeddings of different augmentations. To boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. The designed asymmetric predictor head and an InfoNCE warm-up strategy enhance the robustness to hyper-parameters and benefit the resulting performance. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures, including various lightweight networks (\eg, EfficientNet and MobileNet).

Related papers

• Augmentations vs Algorithms: What Works in Self-Supervised Learning [9.194402355758164]
  We study the relative effects of data augmentations, pretraining algorithms, and model architectures in Self-Supervised Learning (SSL) We propose a new framework which unifies many seemingly disparate SSL methods into a single shared template.
  arXiv  Detail & Related papers  (2024-03-08T23:42:06Z)
• Self-Supervised Representation Learning with Meta Comprehensive Regularization [11.387994024747842]
  We introduce a module called CompMod with Meta Comprehensive Regularization (MCR), embedded into existing self-supervised frameworks. We update our proposed model through a bi-level optimization mechanism, enabling it to capture comprehensive features. We provide theoretical support for our proposed method from information theory and causal counterfactual perspective.
  arXiv  Detail & Related papers  (2024-03-03T15:53:48Z)
• The Common Stability Mechanism behind most Self-Supervised Learning Approaches [64.40701218561921]
  We provide a framework to explain the stability mechanism of different self-supervised learning techniques. We discuss the working mechanism of contrastive techniques like SimCLR, non-contrastive techniques like BYOL, SWAV, SimSiam, Barlow Twins, and DINO. We formulate different hypotheses and test them using the Imagenet100 dataset.
  arXiv  Detail & Related papers  (2024-02-22T20:36:24Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations [30.745749133759304]
  We develop a theoretical framework to analyze the transferability of self-supervised contrastive learning. We show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL)
  arXiv  Detail & Related papers  (2023-03-02T09:26:20Z)
• The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning [62.601681746034956]
  Self-supervised learning (SSL) has emerged as a desirable paradigm in computer vision. We propose a data-driven geometric strategy to analyze different SSL models using local neighborhoods in the feature space induced by each.
  arXiv  Detail & Related papers  (2022-09-18T18:15:38Z)
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation [68.47096022526927]
  Self-supervised learning has achieved great success in learning visual representations without data annotations. We introduce a novel relational SSL paradigm that learns representations by modeling the relationship between different instances. Our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.
  arXiv  Detail & Related papers  (2021-07-20T06:53:07Z)
• Memory-Augmented Relation Network for Few-Shot Learning [114.47866281436829]
  In this work, we investigate a new metric-learning method, Memory-Augmented Relation Network (MRN) In MRN, we choose the samples that are visually similar from the working context, and perform weighted information propagation to attentively aggregate helpful information from chosen ones to enhance its representation. We empirically demonstrate that MRN yields significant improvement over its ancestor and achieves competitive or even better performance when compared with other few-shot learning approaches.
  arXiv  Detail & Related papers  (2020-05-09T10:09:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.