On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning

• URL: http://arxiv.org/abs/2207.10862v1
• Date: Fri, 22 Jul 2022 03:49:50 GMT
• Title: On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning
• Authors: Rohit Gupta, Naveed Akhtar, Ajmal Mian and Mubarak Shah
• Abstract summary: Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. It is still largely unknown if the nature of the representation induced by the two learning paradigms is similar. We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We devise strategies that are simple, yet effective in improving model robustness with CSL training.
• Score: 104.00264962878956
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. However, it is still largely unknown if the nature of the representation induced by the two learning paradigms is similar. We investigate this under the lens of adversarial robustness. Our analytical treatment of the problem reveals intrinsic higher sensitivity of CSL over supervised learning. It identifies the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We establish that this increases model sensitivity to input perturbations in the presence of false negatives in the training data. Our finding is supported by extensive experiments for image and video classification using adversarial perturbations and other input corruptions. Building on the insights, we devise strategies that are simple, yet effective in improving model robustness with CSL training. We demonstrate up to 68% reduction in the performance gap between adversarially attacked CSL and its supervised counterpart. Finally, we contribute to robust CSL paradigm by incorporating our findings in adversarial self-supervised learning. We demonstrate an average gain of about 5% over two different state-of-the-art methods in this domain.

Related papers

• Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling [26.444935219428036]
  Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist. We propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed.
  arXiv  Detail & Related papers  (2023-11-27T22:52:45Z)
• An Investigation of Representation and Allocation Harms in Contrastive Learning [55.42336321517228]
  We demonstrate that contrastive learning (CL) tends to collapse representations of minority groups with certain majority groups. We refer to this phenomenon as representation harm and demonstrate it on image and text datasets using the corresponding popular CL methods. We provide a theoretical explanation for representation harm using a neural block model that leads to a representational collapse in a contrastive learning setting.
  arXiv  Detail & Related papers  (2023-10-02T19:25:37Z)
• InfoNCE Loss Provably Learns Cluster-Preserving Representations [54.28112623495274]
  We show that the representation learned by InfoNCE with a finite number of negative samples is consistent with respect to clusters in the data. Our main result is to show that the representation learned by InfoNCE with a finite number of negative samples is also consistent with respect to clusters in the data.
  arXiv  Detail & Related papers  (2023-02-15T19:45:35Z)
• Self-Supervised Visual Representation Learning via Residual Momentum [15.515169550346517]
  Self-supervised learning (SSL) approaches have shown promising capabilities in learning the representation from unlabeled data. momentum-based SSL frameworks suffer from a large gap in representation between the online encoder (student) and the momentum encoder (teacher) This paper is the first to investigate and identify this invisible gap as a bottleneck that has been overlooked in the existing SSL frameworks. We propose "residual momentum" to directly reduce this gap to encourage the student to learn the representation as close to that of the teacher as possible.
  arXiv  Detail & Related papers  (2022-11-17T19:54:02Z)
• Siamese Prototypical Contrastive Learning [24.794022951873156]
  Contrastive Self-supervised Learning (CSL) is a practical solution that learns meaningful visual representations from massive data in an unsupervised approach. In this paper, we tackle this problem by introducing a simple but effective contrastive learning framework. The key insight is to employ siamese-style metric loss to match intra-prototype features, while increasing the distance between inter-prototype features.
  arXiv  Detail & Related papers  (2022-08-18T13:25:30Z)
• Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness [69.39073806630583]
  Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields. We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
  arXiv  Detail & Related papers  (2022-07-22T06:30:44Z)
• Robustness through Cognitive Dissociation Mitigation in Contrastive Adversarial Training [2.538209532048867]
  We introduce a novel neural network training framework that increases model's adversarial robustness to adversarial attacks. We propose to improve model robustness to adversarial attacks by learning feature representations consistent under both data augmentations and adversarial perturbations. We validate our method on the CIFAR-10 dataset on which it outperforms both robust accuracy and clean accuracy over alternative supervised and self-supervised adversarial learning methods.
  arXiv  Detail & Related papers  (2022-03-16T21:41:27Z)
• How Robust are Discriminatively Trained Zero-Shot Learning Models? [9.62543698736491]
  We present novel analyses on the robustness of discriminative ZSL to image corruptions. We release the first ZSL corruption robustness datasets SUN-C, CUB-C and AWA2-C.
  arXiv  Detail & Related papers  (2022-01-26T14:41:10Z)
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation [68.47096022526927]
  Self-supervised learning has achieved great success in learning visual representations without data annotations. We introduce a novel relational SSL paradigm that learns representations by modeling the relationship between different instances. Our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.
  arXiv  Detail & Related papers  (2021-07-20T06:53:07Z)
• Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations [183.03278932562438]
  This paper presents an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We show that our approach achieves higher efficiency in visual representations and thus delivers a key message to inspire the future research of self-supervised visual representation learning.
  arXiv  Detail & Related papers  (2020-11-19T16:26:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.