PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization

• URL: http://arxiv.org/abs/2310.06182v2
• Date: Sat, 28 Oct 2023 17:01:47 GMT
• Title: PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization
• Authors: Jiancong Xiao, Ruoyu Sun, Zhi- Quan Luo
• Abstract summary: Deep neural networks (DNNs) are vulnerable to adversarial attacks. adversarially robust generalization is crucial in establishing defense algorithms against adversarial attacks. This paper focuses on norm-based perturbation complexity, based on a PAC-Bayes approach.
• Score: 25.272738030198862
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) are vulnerable to adversarial attacks. It is found empirically that adversarially robust generalization is crucial in establishing defense algorithms against adversarial attacks. Therefore, it is interesting to study the theoretical guarantee of robust generalization. This paper focuses on norm-based complexity, based on a PAC-Bayes approach (Neyshabur et al., 2017). The main challenge lies in extending the key ingredient, which is a weight perturbation bound in standard settings, to the robust settings. Existing attempts heavily rely on additional strong assumptions, leading to loose bounds. In this paper, we address this issue and provide a spectrally-normalized robust generalization bound for DNNs. Compared to existing bounds, our bound offers two significant advantages: Firstly, it does not depend on additional assumptions. Secondly, it is considerably tighter, aligning with the bounds of standard generalization. Therefore, our result provides a different perspective on understanding robust generalization: The mismatch terms between standard and robust generalization bounds shown in previous studies do not contribute to the poor robust generalization. Instead, these disparities solely due to mathematical issues. Finally, we extend the main result to adversarial robustness against general non-$\ell_p$ attacks and other neural network architectures.

Related papers

• Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization [29.044914673801856]
  Training Deep Neural Networks (DNNs) with adversarial examples often results in poor generalization to test-time adversarial data. This paper investigates this issue through the lens of Rademacher complexity. We aim to construct a new cover that possesses two properties: 1) compatibility with adversarial examples, and 2) precision comparable to covers used in standard settings.
  arXiv  Detail & Related papers  (2024-06-08T06:45:19Z)
• PAC-Bayesian Adversarially Robust Generalization Bounds for Graph Neural Network [5.340644246815989]
  Graph neural networks (GNNs) are vulnerable to adversarial attacks. In this paper, we provide adversarially robust generalization bounds for two kinds of popular GNNs.
  arXiv  Detail & Related papers  (2024-02-06T14:34:17Z)
• Enhance Diffusion to Improve Robust Generalization [39.9012723077658]
  emphAdversarial Training (AT) is one of the strongest defense mechanisms against adversarial perturbations. This paper focuses on the primary AT framework - Projected Gradient Descent Adversarial Training (PGD-AT) We propose a novel approach, emphDiffusion Enhanced Adversarial Training (DEAT), to manipulate the diffusion term to improve robust generalization with virtually no extra computational burden.
  arXiv  Detail & Related papers  (2023-06-05T06:36:18Z)
• PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization [48.26492774959634]
  We develop a compression approach based on quantizing neural network parameters in a linear subspace. We find large models can be compressed to a much greater extent than previously known, encapsulating Occam's razor.
  arXiv  Detail & Related papers  (2022-11-24T13:50:16Z)
• On the Importance of Gradient Norm in PAC-Bayesian Bounds [92.82627080794491]
  We propose a new generalization bound that exploits the contractivity of the log-Sobolev inequalities. We empirically analyze the effect of this new loss-gradient norm term on different neural architectures.
  arXiv  Detail & Related papers  (2022-10-12T12:49:20Z)
• Robustness Implies Generalization via Data-Dependent Generalization Bounds [24.413499775513145]
  This paper proves that robustness implies generalization via data-dependent generalization bounds. We present several examples, including ones for lasso and deep learning, in which our bounds are provably preferable.
  arXiv  Detail & Related papers  (2022-06-27T17:58:06Z)
• Understanding Robust Generalization in Learning Regular Languages [85.95124524975202]
  We study robust generalization in the context of using recurrent neural networks to learn regular languages. We propose a compositional strategy to address this. We theoretically prove that the compositional strategy generalizes significantly better than the end-to-end strategy.
  arXiv  Detail & Related papers  (2022-02-20T02:50:09Z)
• Interpolation can hurt robust generalization even when there is no noise [76.3492338989419]
  We show that avoiding generalization through ridge regularization can significantly improve generalization even in the absence of noise. We prove this phenomenon for the robust risk of both linear regression and classification and hence provide the first theoretical result on robust overfitting.
  arXiv  Detail & Related papers  (2021-08-05T23:04:15Z)
• Robustness, Privacy, and Generalization of Adversarial Training [84.38148845727446]
  This paper establishes and quantifies the privacy-robustness trade-off and generalization-robustness trade-off in adversarial training. We show that adversarial training is $(varepsilon, delta)$-differentially private, where the magnitude of the differential privacy has a positive correlation with the robustified intensity. Our generalization bounds do not explicitly rely on the parameter size which would be large in deep learning.
  arXiv  Detail & Related papers  (2020-12-25T13:35:02Z)
• In Search of Robust Measures of Generalization [79.75709926309703]
  We develop bounds on generalization error, optimization error, and excess risk. When evaluated empirically, most of these bounds are numerically vacuous. We argue that generalization measures should instead be evaluated within the framework of distributional robustness.
  arXiv  Detail & Related papers  (2020-10-22T17:54:25Z)
• On the generalization of bayesian deep nets for multi-class classification [27.39403411896995]
  We propose a new generalization bound for Bayesian deep nets by exploiting the contractivity of the Log-Sobolev inequalities. Using these inequalities adds an additional loss-gradient norm term to the generalization bound, which is intuitively a surrogate of the model complexity.
  arXiv  Detail & Related papers  (2020-02-23T09:05:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.