On the Convergence of Certified Robust Training with Interval Bound Propagation

• URL: http://arxiv.org/abs/2203.08961v1
• Date: Wed, 16 Mar 2022 21:49:13 GMT
• Title: On the Convergence of Certified Robust Training with Interval Bound Propagation
• Authors: Yihan Wang, Zhouxing Shi, Quanquan Gu, Cho-Jui Hsieh
• Abstract summary: We present a theoretical analysis on the convergence of IBP training. We show that when using IBP training to train a randomly two-layer ReLU neural network with logistic loss, gradient descent can linearly converge to zero robust training error.
• Score: 147.77638840942447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interval Bound Propagation (IBP) is so far the base of state-of-the-art methods for training neural networks with certifiable robustness guarantees when potential adversarial perturbations present, while the convergence of IBP training remains unknown in existing literature. In this paper, we present a theoretical analysis on the convergence of IBP training. With an overparameterized assumption, we analyze the convergence of IBP robust training. We show that when using IBP training to train a randomly initialized two-layer ReLU neural network with logistic loss, gradient descent can linearly converge to zero robust training error with a high probability if we have sufficiently small perturbation radius and large network width.

Related papers

• Evolutionary algorithms as an alternative to backpropagation for supervised training of Biophysical Neural Networks and Neural ODEs [12.357635939839696]
  We investigate the use of "gradient-estimating" evolutionary algorithms for training biophysically based neural networks. We find that EAs have several advantages making them desirable over direct BP. Our findings suggest that biophysical neurons could provide useful benchmarks for testing the limits of BP methods.
  arXiv  Detail & Related papers  (2023-11-17T20:59:57Z)
• Understanding Certified Training with Interval Bound Propagation [6.688598900034783]
  Training certifiably robust neural networks is becoming more relevant. We show that training methods based on the imprecise interval bound propagation (IBP) consistently outperform those leveraging more precise bounding methods. This hints at the existence of new training methods that do not induce the strong regularization required for tight IBP bounds.
  arXiv  Detail & Related papers  (2023-06-17T21:13:30Z)
• A Theoretical Framework for Inference and Learning in Predictive Coding Networks [41.58529335439799]
  Predictive coding (PC) is an influential theory in computational neuroscience. We provide a comprehensive theoretical analysis of the properties of PCNs trained with prospective configuration.
  arXiv  Detail & Related papers  (2022-07-21T04:17:55Z)
• IBP Regularization for Verified Adversarial Robustness via Branch-and-Bound [85.6899802468343]
  We present IBP-R, a novel verified training algorithm that is both simple effective. We also present UPB, a novel robustness based on $beta$-CROWN, that reduces the cost state-of-the-art branching algorithms.
  arXiv  Detail & Related papers  (2022-06-29T17:13:25Z)
• A Theoretical View of Linear Backpropagation and Its Convergence [55.69505060636719]
  Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs) Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks. We provide theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training.
  arXiv  Detail & Related papers  (2021-12-21T07:18:00Z)
• Towards Evaluating and Training Verifiably Robust Neural Networks [81.39994285743555]
  We study the relationship between IBP and CROWN, and prove that CROWN is always tighter than IBP when choosing appropriate bounding lines. We propose a relaxed version of CROWN, linear bound propagation (LBP), that can be used to verify large networks to obtain lower verified errors.
  arXiv  Detail & Related papers  (2021-04-01T13:03:48Z)
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks [56.084798078072396]
  We take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time. We show that convergence to a global minimum is guaranteed for networks with quadratic widths in the sample size and linear in their depth at a time logarithmic in both. Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size.
  arXiv  Detail & Related papers  (2021-01-12T00:40:45Z)
• Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias [65.13042449121411]
  In practice, training a network with the gradient estimates provided by EP does not scale to visual tasks harder than MNIST. We show that a bias in the gradient estimate of EP, inherent in the use of finite nudging, is responsible for this phenomenon. We apply these techniques to train an architecture with asymmetric forward and backward connections, yielding a 13.2% test error.
  arXiv  Detail & Related papers  (2020-06-06T09:36:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.