Understanding and Achieving Efficient Robustness with Adversarial Contrastive Learning

• URL: http://arxiv.org/abs/2101.10027v1
• Date: Mon, 25 Jan 2021 11:57:52 GMT
• Title: Understanding and Achieving Efficient Robustness with Adversarial Contrastive Learning
• Authors: Anh Bui, Trung Le, He Zhao, Paul Montague, Seyit Camtepe, Dinh Phung
• Abstract summary: Adversarial Supervised Contrastive Learning (ASCL) approach outperforms the state-of-the-art defenses by $2.6%$ in terms of the robust accuracy. Our ASCL with the proposed selection strategy can further gain $1.4%$ improvement with only $42.8%$ positives and $6.3%$ negatives compared with ASCL without a selection strategy.
• Score: 34.97017489872795
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Contrastive learning (CL) has recently emerged as an effective approach to learning representation in a range of downstream tasks. Central to this approach is the selection of positive (similar) and negative (dissimilar) sets to provide the model the opportunity to `contrast' between data and class representation in the latent space. In this paper, we investigate CL for improving model robustness using adversarial samples. We first designed and performed a comprehensive study to understand how adversarial vulnerability behaves in the latent space. Based on these empirical evidences, we propose an effective and efficient supervised contrastive learning to achieve model robustness against adversarial attacks. Moreover, we propose a new sample selection strategy that optimizes the positive/negative sets by removing redundancy and improving correlation with the anchor. Experiments conducted on benchmark datasets show that our Adversarial Supervised Contrastive Learning (ASCL) approach outperforms the state-of-the-art defenses by $2.6\%$ in terms of the robust accuracy, whilst our ASCL with the proposed selection strategy can further gain $1.4\%$ improvement with only $42.8\%$ positives and $6.3\%$ negatives compared with ASCL without a selection strategy.

Related papers

• Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learning [21.018893978967053]
  In-Context Learning (ICL) is sensitive to the choice, order, and verbaliser used to encode the demonstrations in the prompt. Retrieval-Augmented ICL methods try to address this problem by leveraging retrievers to extract semantically related examples as demonstrations. Our study reveals that retrieval-augmented models can enhance robustness against test sample attacks. We introduce an effective training-free adversarial defence method, DARD, which enriches the example pool with those attacked samples.
  arXiv  Detail & Related papers  (2024-05-24T23:56:36Z)
• Cross-Stream Contrastive Learning for Self-Supervised Skeleton-Based Action Recognition [22.067143671631303]
  Self-supervised skeleton-based action recognition enjoys a rapid growth along with the development of contrastive learning. We propose a Cross-Stream Contrastive Learning framework for skeleton-based action Representation learning (CSCLR) Specifically, the proposed CSCLR not only utilizes intra-stream contrast pairs, but introduces inter-stream contrast pairs as hard samples to formulate a better representation learning.
  arXiv  Detail & Related papers  (2023-05-03T10:31:35Z)
• Cluster-aware Contrastive Learning for Unsupervised Out-of-distribution Detection [0.0]
  Unsupervised out-of-distribution (OOD) Detection aims to separate the samples falling outside the distribution of training data without label information. We propose Cluster-aware Contrastive Learning (CCL) framework for unsupervised OOD detection, which considers both instance-level and semantic-level information.
  arXiv  Detail & Related papers  (2023-02-06T07:21:03Z)
• Effective Targeted Attacks for Adversarial Self-Supervised Learning [58.14233572578723]
  unsupervised adversarial training (AT) has been highlighted as a means of achieving robustness in models without any label information. We propose a novel positive mining for targeted adversarial attack to generate effective adversaries for adversarial SSL frameworks. Our method demonstrates significant enhancements in robustness when applied to non-contrastive SSL frameworks, and less but consistent robustness improvements with contrastive SSL frameworks.
  arXiv  Detail & Related papers  (2022-10-19T11:43:39Z)
• Adversarial Contrastive Learning via Asymmetric InfoNCE [64.42740292752069]
  We propose to treat adversarial samples unequally when contrasted with an asymmetric InfoNCE objective. In the asymmetric fashion, the adverse impacts of conflicting objectives between CL and adversarial learning can be effectively mitigated. Experiments show that our approach consistently outperforms existing Adversarial CL methods.
  arXiv  Detail & Related papers  (2022-07-18T04:14:36Z)
• Improving Gradient-based Adversarial Training for Text Classification by Contrastive Learning and Auto-Encoder [18.375585982984845]
  We focus on enhancing the model's ability to defend gradient-based adversarial attack during the model's training process. We propose two novel adversarial training approaches: CARL and RAR. Experiments show that the proposed two approaches outperform strong baselines on various text classification datasets.
  arXiv  Detail & Related papers  (2021-09-14T09:08:58Z)
• Semi-supervised Contrastive Learning with Similarity Co-calibration [72.38187308270135]
  We propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL) SsCL combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning. We show that SsCL produces more discriminative representation and is beneficial to few shot learning.
  arXiv  Detail & Related papers  (2021-05-16T09:13:56Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.