A Theoretical View of Linear Backpropagation and Its Convergence

• URL: http://arxiv.org/abs/2112.11018v2
• Date: Wed, 10 Jan 2024 12:25:26 GMT
• Title: A Theoretical View of Linear Backpropagation and Its Convergence
• Authors: Ziang Li, Yiwen Guo, Haodi Liu, and Changshui Zhang
• Abstract summary: Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs) Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks. We provide theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training.
• Score: 55.69505060636719
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments.

Related papers

• A Theoretical Framework for Inference and Learning in Predictive Coding Networks [41.58529335439799]
  Predictive coding (PC) is an influential theory in computational neuroscience. We provide a comprehensive theoretical analysis of the properties of PCNs trained with prospective configuration.
  arXiv  Detail & Related papers  (2022-07-21T04:17:55Z)
• Constrained Parameter Inference as a Principle for Learning [5.080518039966762]
  We propose constrained parameter inference (COPI) as a new principle for learning. COPI allows for the estimation of network parameters under the constraints of decorrelated neural inputs and top-down perturbations of neural states. We show that COPI not only is more biologically plausible but also provides distinct advantages for fast learning, compared with standard backpropagation of error.
  arXiv  Detail & Related papers  (2022-03-22T13:40:57Z)
• On the Convergence of Certified Robust Training with Interval Bound Propagation [147.77638840942447]
  We present a theoretical analysis on the convergence of IBP training. We show that when using IBP training to train a randomly two-layer ReLU neural network with logistic loss, gradient descent can linearly converge to zero robust training error.
  arXiv  Detail & Related papers  (2022-03-16T21:49:13Z)
• Towards Evaluating and Training Verifiably Robust Neural Networks [81.39994285743555]
  We study the relationship between IBP and CROWN, and prove that CROWN is always tighter than IBP when choosing appropriate bounding lines. We propose a relaxed version of CROWN, linear bound propagation (LBP), that can be used to verify large networks to obtain lower verified errors.
  arXiv  Detail & Related papers  (2021-04-01T13:03:48Z)
• Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks [40.51949948934705]
  Predictive coding networks (PCNs) are an influential model for information processing in the brain. BP is commonly regarded to be the most successful learning method in modern machine learning. We show that a biologically plausible algorithm is able to exactly replicate the accuracy of BP on complex architectures.
  arXiv  Detail & Related papers  (2021-03-05T14:57:01Z)
• Belief Propagation Neural Networks [103.97004780313105]
  We introduce belief propagation neural networks (BPNNs) BPNNs operate on factor graphs and generalize Belief propagation (BP) We show that BPNNs converges 1.7x faster on Ising models while providing tighter bounds. On challenging model counting problems, BPNNs compute estimates 100's of times faster than state-of-the-art handcrafted methods.
  arXiv  Detail & Related papers  (2020-07-01T07:39:51Z)
• A Theoretical Framework for Target Propagation [75.52598682467817]
  We analyze target propagation (TP), a popular but not yet fully understood alternative to backpropagation (BP) Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training.
  arXiv  Detail & Related papers  (2020-06-25T12:07:06Z)
• Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems [83.98774574197613]
  We take one of the simplest inference methods, a truncated max-product Belief propagation, and add what is necessary to make it a proper component of a deep learning model. This BP-Layer can be used as the final or an intermediate block in convolutional neural networks (CNNs) The model is applicable to a range of dense prediction problems, is well-trainable and provides parameter-efficient and robust solutions in stereo, optical flow and semantic segmentation.
  arXiv  Detail & Related papers  (2020-03-13T13:11:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.