Practical Convex Formulation of Robust One-hidden-layer Neural Network Training

• URL: http://arxiv.org/abs/2105.12237v1
• Date: Tue, 25 May 2021 22:06:27 GMT
• Title: Practical Convex Formulation of Robust One-hidden-layer Neural Network Training
• Authors: Yatong Bai, Tanmay Gautam, Yu Gai, Somayeh Sojoudi
• Abstract summary: We show that the training of a one-hidden-layer, scalar-output fully-connected ReLULU neural network can be reformulated as a finite-dimensional convex program. We derive a convex optimization approach to efficiently solve the "adversarial training" problem. Our method can be applied to binary classification and regression, and provides an alternative to the current adversarial training methods.
• Score: 12.71266194474117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has shown that the training of a one-hidden-layer, scalar-output fully-connected ReLU neural network can be reformulated as a finite-dimensional convex program. Unfortunately, the scale of such a convex program grows exponentially in data size. In this work, we prove that a stochastic procedure with a linear complexity well approximates the exact formulation. Moreover, we derive a convex optimization approach to efficiently solve the "adversarial training" problem, which trains neural networks that are robust to adversarial input perturbations. Our method can be applied to binary classification and regression, and provides an alternative to the current adversarial training methods, such as Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). We demonstrate in experiments that the proposed method achieves a noticeably better adversarial robustness and performance than the existing methods.

Related papers

• Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training [30.452060061499523]
  We introduce an approximation technique for the likelihood ratio (LR) method to alleviate computational and memory demands in gradient estimation. Experiments demonstrate the effectiveness of the approximation technique in neural network training.
  arXiv  Detail & Related papers  (2024-03-18T23:23:50Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Robust Stochastically-Descending Unrolled Networks [85.6993263983062]
  Deep unrolling is an emerging learning-to-optimize method that unrolls a truncated iterative algorithm in the layers of a trainable neural network. We show that convergence guarantees and generalizability of the unrolled networks are still open theoretical problems. We numerically assess unrolled architectures trained under the proposed constraints in two different applications.
  arXiv  Detail & Related papers  (2023-12-25T18:51:23Z)
• The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models [75.33431791218302]
  Deep Neural Network Network (DNN) models are used for programming purposes. In this paper we examine the use of convex neural recovery models. We show that all the stationary non-dimensional objective objective can be characterized as the standard a global subsampled convex solvers program. We also show that all the stationary non-dimensional objective objective can be characterized as the standard a global subsampled convex solvers program.
  arXiv  Detail & Related papers  (2023-12-19T23:04:56Z)
• Stochastic Unrolled Federated Learning [85.6993263983062]
  We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning. Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
  arXiv  Detail & Related papers  (2023-05-24T17:26:22Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Robust Explanation Constraints for Neural Networks [33.14373978947437]
  Post-hoc explanation methods used with the intent of neural networks are sometimes said to help engender trust in their outputs. Our training method is the only method able to learn neural networks with insights about robustness tested across all six tested networks.
  arXiv  Detail & Related papers  (2022-12-16T14:40:25Z)
• Deep unfolding as iterative regularization for imaging inverse problems [6.485466095579992]
  Deep unfolding methods guide the design of deep neural networks (DNNs) through iterative algorithms. We prove that the unfolded DNN will converge to it stably. We demonstrate with an example of MRI reconstruction that the proposed method outperforms conventional unfolding methods.
  arXiv  Detail & Related papers  (2022-11-24T07:38:47Z)
• Adaptive Learning Rate and Momentum for Training Deep Neural Networks [0.0]
  We develop a fast training method motivated by the nonlinear Conjugate Gradient (CG) framework. Experiments in image classification datasets show that our method yields faster convergence than other local solvers.
  arXiv  Detail & Related papers  (2021-06-22T05:06:56Z)
• Deep Magnification-Flexible Upsampling over 3D Point Clouds [103.09504572409449]
  We propose a novel end-to-end learning-based framework to generate dense point clouds. We first formulate the problem explicitly, which boils down to determining the weights and high-order approximation errors. Then, we design a lightweight neural network to adaptively learn unified and sorted weights as well as the high-order refinements.
  arXiv  Detail & Related papers  (2020-11-25T14:00:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.