Adversarial Training from Mean Field Perspective

• URL: http://arxiv.org/abs/2505.14021v1
• Date: Tue, 20 May 2025 07:22:21 GMT
• Title: Adversarial Training from Mean Field Perspective
• Authors: Soichiro Kumano, Hiroshi Kera, Toshihiko Yamasaki,
• Abstract summary: We present the first theoretical analysis of adversarial training in random deep neural networks without any assumptions on data distributions.<n>We derive (empirically tight) upper bounds of $ell_q$ norm-based adversarial loss with $ell_p$ norm-based adversarial examples.<n>We prove that networks without shortcuts are generally not adversarially trainable and that adversarial training reduces network capacity.
• Score: 27.368408524000778
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Although adversarial training is known to be effective against adversarial examples, training dynamics are not well understood. In this study, we present the first theoretical analysis of adversarial training in random deep neural networks without any assumptions on data distributions. We introduce a new theoretical framework based on mean field theory, which addresses the limitations of existing mean field-based approaches. Based on this framework, we derive (empirically tight) upper bounds of $\ell_q$ norm-based adversarial loss with $\ell_p$ norm-based adversarial examples for various values of $p$ and $q$. Moreover, we prove that networks without shortcuts are generally not adversarially trainable and that adversarial training reduces network capacity. We also show that network width alleviates these issues. Furthermore, we present the various impacts of the input and output dimensions on the upper bounds and time evolution of the weight variance.

Related papers

• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• Generalization Bounds for Adversarial Contrastive Learning [10.893632710192016]
  We use Rademacher complexity to analyze the generalization performance of ACL. Our theory shows that the average adversarial risk of the downstream tasks can be upper bounded by the adversarial unsupervised risk of the upstream task.
  arXiv  Detail & Related papers  (2023-02-21T12:44:59Z)
• Chaos Theory and Adversarial Robustness [0.0]
  This paper uses ideas from Chaos Theory to explain, analyze, and quantify the degree to which neural networks are susceptible to or robust against adversarial attacks. We present a new metric, the "susceptibility ratio," given by $hat Psi(h, theta)$, which captures how greatly a model's output will be changed by perturbations to a given input.
  arXiv  Detail & Related papers  (2022-10-20T03:39:44Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)
• Sparsity Winning Twice: Better Robust Generalization from More Efficient Training [94.92954973680914]
  We introduce two alternatives for sparse adversarial training: (i) static sparsity and (ii) dynamic sparsity. We find both methods to yield win-win: substantially shrinking the robust generalization gap and alleviating the robust overfitting. Our approaches can be combined with existing regularizers, establishing new state-of-the-art results in adversarial training.
  arXiv  Detail & Related papers  (2022-02-20T15:52:08Z)
• Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach [60.67748036747221]
  Implicit neural networks offer competitive performance and reduced memory consumption. They can remain brittle with respect to input adversarial perturbations. This paper proposes a theoretical and computational framework for robustness verification of implicit neural networks.
  arXiv  Detail & Related papers  (2021-12-10T03:08:55Z)
• Adversarial Robustness with Semi-Infinite Constrained Learning [177.42714838799924]
  Deep learning to inputs perturbations has raised serious questions about its use in safety-critical domains. We propose a hybrid Langevin Monte Carlo training approach to mitigate this issue. We show that our approach can mitigate the trade-off between state-of-the-art performance and robust robustness.
  arXiv  Detail & Related papers  (2021-10-29T13:30:42Z)
• Adversarial Robustness Guarantees for Random Deep Neural Networks [15.68430580530443]
  adversarial examples are incorrectly classified inputs that are extremely close to a correctly classified input. We prove that for any $pge1$, the $ellp$ distance of any given input from the classification boundary scales as one over the square root of the dimension of the input times the $ellp$ norm of the input. The results constitute a fundamental advance in the theoretical understanding of adversarial examples, and open the way to a thorough theoretical characterization of the relation between network architecture and robustness to adversarial perturbations.
  arXiv  Detail & Related papers  (2020-04-13T13:07:26Z)
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training [78.30684998080346]
  We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process. As a practical application of our findings, we demonstrate that such sub-networks can help in cutting down the total time of adversarial training.
  arXiv  Detail & Related papers  (2020-03-06T03:11:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.