Overcoming the Paradox of Certified Training with Gaussian Smoothing

• URL: http://arxiv.org/abs/2403.07095v2
• Date: Tue, 25 Jun 2024 13:46:24 GMT
• Title: Overcoming the Paradox of Certified Training with Gaussian Smoothing
• Authors: Stefan Balauca, Mark Niklas Müller, Yuhao Mao, Maximilian Baader, Marc Fischer, Martin Vechev,
• Abstract summary: Training neural networks with high certified accuracy against adversarial examples remains an open problem. We show theoretically that Gaussian Loss Smoothing can alleviate both issues. Our results clearly demonstrate the promise of Gaussian Loss Smoothing for training certifiably robust neural networks.
• Score: 14.061189994638667
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training neural networks with high certified accuracy against adversarial examples remains an open problem despite significant efforts. While certification methods can effectively leverage tight convex relaxations for bound computation, in training, these methods perform worse than looser relaxations. Prior work hypothesized that this is caused by the discontinuity and perturbation sensitivity of the loss surface induced by these tighter relaxations. In this work, we show theoretically that Gaussian Loss Smoothing can alleviate both issues. We confirm this empirically by proposing a certified training method combining PGPE, an algorithm computing gradients of a smoothed loss, with different convex relaxations. When using this training method, we observe that tighter bounds indeed lead to strictly better networks. While scaling PGPE training remains challenging due to high computational cost, we show that by using a not theoretically sound, yet much cheaper smoothing approximation, we obtain better certified accuracies than state-of-the-art methods when training on the same network architecture. Our results clearly demonstrate the promise of Gaussian Loss Smoothing for training certifiably robust neural networks.

Related papers

• The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information [35.34142909458158]
  We show that we can leverage curvature information in OBS-like fashion upon the projection step of classic iterative sparse recovery algorithms such as IHT. We present extensions of this approach to the practical task of obtaining accurate sparses, and validate it experimentally at scale for Transformer-based models on vision and language tasks.
  arXiv  Detail & Related papers  (2024-08-30T10:06:26Z)
• Robust Stochastically-Descending Unrolled Networks [85.6993263983062]
  Deep unrolling is an emerging learning-to-optimize method that unrolls a truncated iterative algorithm in the layers of a trainable neural network. We show that convergence guarantees and generalizability of the unrolled networks are still open theoretical problems. We numerically assess unrolled architectures trained under the proposed constraints in two different applications.
  arXiv  Detail & Related papers  (2023-12-25T18:51:23Z)
• Resilient Constrained Learning [94.27081585149836]
  This paper presents a constrained learning approach that adapts the requirements while simultaneously solving the learning task. We call this approach resilient constrained learning after the term used to describe ecological systems that adapt to disruptions by modifying their operation.
  arXiv  Detail & Related papers  (2023-06-04T18:14:18Z)
• Towards Scaling Difference Target Propagation by Learning Backprop Targets [64.90165892557776]
  Difference Target Propagation is a biologically-plausible learning algorithm with close relation with Gauss-Newton (GN) optimization. We propose a novel feedback weight training scheme that ensures both that DTP approximates BP and that layer-wise feedback weight training can be restored. We report the best performance ever achieved by DTP on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2022-01-31T18:20:43Z)
• DeepSplit: Scalable Verification of Deep Neural Networks via Operator Splitting [70.62923754433461]
  Analyzing the worst-case performance of deep neural networks against input perturbations amounts to solving a large-scale non- optimization problem. We propose a novel method that can directly solve a convex relaxation of the problem to high accuracy, by splitting it into smaller subproblems that often have analytical solutions.
  arXiv  Detail & Related papers  (2021-06-16T20:43:49Z)
• Feature Purification: How Adversarial Training Performs Robust Deep Learning [66.05472746340142]
  We show a principle that we call Feature Purification, where we show one of the causes of the existence of adversarial examples is the accumulation of certain small dense mixtures in the hidden weights during the training process of a neural network. We present both experiments on the CIFAR-10 dataset to illustrate this principle, and a theoretical result proving that for certain natural classification tasks, training a two-layer neural network with ReLU activation using randomly gradient descent indeed this principle.
  arXiv  Detail & Related papers  (2020-05-20T16:56:08Z)
• Tightened Convex Relaxations for Neural Network Robustness Certification [10.68833097448566]
  We exploit the structure of ReLU networks to improve relaxation errors through a novel partition-based certification procedure. The proposed method is proven to tighten existing linear programming relaxations, and achieves zero relaxation error as the result is made finer.
  arXiv  Detail & Related papers  (2020-04-01T16:59:21Z)
• Explicitly Trained Spiking Sparsity in Spiking Neural Networks with Backpropagation [7.952659059689134]
  Spiking Neural Networks (SNNs) are being explored for their potential energy efficiency resulting from sparse, event-driven computations. We propose an explicit inclusion of spike counts in the loss function, along with a traditional error loss, to optimize weight parameters for both accuracy and spiking sparsity.
  arXiv  Detail & Related papers  (2020-03-02T23:39:18Z)
• Improving the Tightness of Convex Relaxation Bounds for Training Certifiably Robust Classifiers [72.56180590447835]
  Convex relaxations are effective for certifying training and neural networks against norm-bounded adversarial attacks, but they leave a large gap between certifiable and empirical robustness. We propose two experiments that can be used to train neural networks that can be trained in higher certified accuracy than non-regularized baselines.
  arXiv  Detail & Related papers  (2020-02-22T20:19:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.