Generalization Bounds for Adversarial Contrastive Learning
- URL: http://arxiv.org/abs/2302.10633v1
- Date: Tue, 21 Feb 2023 12:44:59 GMT
- Title: Generalization Bounds for Adversarial Contrastive Learning
- Authors: Xin Zou, Weiwei Liu
- Abstract summary: We use Rademacher complexity to analyze the generalization performance of ACL.
Our theory shows that the average adversarial risk of the downstream tasks can be upper bounded by the adversarial unsupervised risk of the upstream task.
- Score: 10.893632710192016
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep networks are well-known to be fragile to adversarial attacks, and
adversarial training is one of the most popular methods used to train a robust
model. To take advantage of unlabeled data, recent works have applied
adversarial training to contrastive learning (Adversarial Contrastive Learning;
ACL for short) and obtain promising robust performance. However, the theory of
ACL is not well understood. To fill this gap, we leverage the Rademacher
complexity to analyze the generalization performance of ACL, with a particular
focus on linear models and multi-layer neural networks under $\ell_p$ attack
($p \ge 1$). Our theory shows that the average adversarial risk of the
downstream tasks can be upper bounded by the adversarial unsupervised risk of
the upstream task. The experimental results validate our theory.
Related papers
- Boosting Adversarial Training via Fisher-Rao Norm-based Regularization [9.975998980413301]
We propose a novel regularization framework, called Logit-Oriented Adversarial Training (LOAT), which can mitigate the trade-off between robustness and accuracy.
Our experiments demonstrate that the proposed regularization strategy can boost the performance of the prevalent adversarial training algorithms.
arXiv Detail & Related papers (2024-03-26T09:22:37Z) - Post-Training Overfitting Mitigation in DNN Classifiers [31.513866929577336]
We show that post-training MM-based regularization substantially mitigates non-malicious overfitting due to class imbalances and overtraining.
Unlike adversarial training, which provides some resilience against attacks but which harms clean (attack-free) generalization, we demonstrate an approach originating from adversarial learning.
arXiv Detail & Related papers (2023-09-28T20:16:24Z) - Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
We propose a novel doubly-robust instance reweighted adversarial framework.
Our importance weights are obtained by optimizing the KL-divergence regularized loss function.
Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
arXiv Detail & Related papers (2023-08-01T06:16:18Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - Layer-wise Regularized Adversarial Training using Layers Sustainability
Analysis (LSA) framework [8.701566919381223]
An appropriate solution to adversarial attacks is adversarial training, which reaches a trade-off between robustness and generalization.
This paper introduces a novel framework (Layer Sustainability Analysis) for the analysis of layer vulnerability in a given neural network in the scenario of adversarial attacks.
The proposed idea performs well theoretically and experimentally for state-of-the-art multilayer perceptron and convolutional neural network architectures.
arXiv Detail & Related papers (2022-02-05T20:05:53Z) - Learning and Certification under Instance-targeted Poisoning [49.55596073963654]
We study PAC learnability and certification under instance-targeted poisoning attacks.
We show that when the budget of the adversary scales sublinearly with the sample complexity, PAC learnability and certification are achievable.
We empirically study the robustness of K nearest neighbour, logistic regression, multi-layer perceptron, and convolutional neural network on real data sets.
arXiv Detail & Related papers (2021-05-18T17:48:15Z) - On the Generalization Properties of Adversarial Training [21.79888306754263]
This paper studies the generalization performance of a generic adversarial training algorithm.
A series of numerical studies are conducted to demonstrate how the smoothness and L1 penalization help improve the adversarial robustness of models.
arXiv Detail & Related papers (2020-08-15T02:32:09Z) - Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions.
We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples.
We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
arXiv Detail & Related papers (2020-06-13T08:24:33Z) - Overfitting in adversarially robust deep learning [86.11788847990783]
We show that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training.
We also show that effects such as the double descent curve do still occur in adversarially trained models, yet fail to explain the observed overfitting.
arXiv Detail & Related papers (2020-02-26T15:40:50Z) - Adversarial Distributional Training for Robust Deep Learning [53.300984501078126]
Adversarial training (AT) is among the most effective techniques to improve model robustness by augmenting training data with adversarial examples.
Most existing AT methods adopt a specific attack to craft adversarial examples, leading to the unreliable robustness against other unseen attacks.
In this paper, we introduce adversarial distributional training (ADT), a novel framework for learning robust models.
arXiv Detail & Related papers (2020-02-14T12:36:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.