When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?

• URL: http://arxiv.org/abs/2102.04998v1
• Date: Tue, 9 Feb 2021 18:04:37 GMT
• Title: When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
• Authors: Niladri S. Chatterji, Philip M. Long, Peter L. Bartlett
• Abstract summary: We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU.
• Score: 51.1848572349154
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero, and prove bounds on the rate of convergence. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU, proposed in previous applied work. We provide two sufficient conditions for convergence. The first is simply a bound on the loss at initialization. The second is a data separation condition used in prior analyses.

Related papers

• On the Stability of Gradient Descent for Large Learning Rate [62.19241612132701]
  Edge of Stability (EoS) has been observed in neural networks training, characterized by a non-monotonic decrease of the loss function over epochs. We show that linear neural networks optimized under a quadratic loss function satisfy the first assumption and also a necessary condition for the second assumption.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks [53.95175206863992]
  We study the type of solutions to which gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. We prove that although shallow ReLU networks are universal approximators, stable shallow networks are not.
  arXiv  Detail & Related papers  (2023-06-30T09:17:39Z)
• Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data [63.34506218832164]
  In this work, we investigate the implicit bias of gradient flow and gradient descent in two-layer fully-connected neural networks with ReLU activations. For gradient flow, we leverage recent work on the implicit bias for homogeneous neural networks to show that leakyally, gradient flow produces a neural network with rank at most two. For gradient descent, provided the random variance is small enough, we show that a single step of gradient descent suffices to drastically reduce the rank of the network, and that the rank remains small throughout training.
  arXiv  Detail & Related papers  (2022-10-13T15:09:54Z)
• Gradient descent provably escapes saddle points in the training of shallow ReLU networks [3.0079490585515343]
  We prove a variant of the relevant dynamical systems result, a center-stable manifold theorem, in which we relax some regularity requirements. Building on a classification of critical points of the square integral loss of shallow ReLU networks measured against an affine target function, we deduce that gradient descent avoids most saddle points.
  arXiv  Detail & Related papers  (2022-08-03T14:08:52Z)
• Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networks [1.14219428942199]
  We study the overparametrization bounds required for the global convergence of gradient descent algorithm for a class of one hidden layer feed-forward neural networks.
  arXiv  Detail & Related papers  (2022-01-28T11:30:06Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• When does gradient descent with logistic loss find interpolating two-layer networks? [51.1848572349154]
  We show that gradient descent drives the training loss to zero if the initial loss is small enough. When the data satisfies certain cluster and separation conditions and the network is wide enough, we show that one step of gradient descent reduces the loss sufficiently that the first result applies.
  arXiv  Detail & Related papers  (2020-12-04T05:16:51Z)
• Asymptotic convergence rate of Dropout on shallow linear neural networks [0.0]
  We analyze the convergence on objective functions induced by Dropout and Dropconnect, when applying them to shallow linear Neural Networks. We obtain a local convergence proof of the gradient flow and a bound on the rate that depends on the data, the rate probability, and the width of the NN.
  arXiv  Detail & Related papers  (2020-12-01T19:02:37Z)
• How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer [5.969858080492586]
  We consider one dimensional (shallow) ReLU neural networks in which weights are chosen randomly and only the terminal layer is trained. We show that for such networks L2-regularized regression corresponds in function space to regularizing the estimate's second derivative for fairly general loss functionals.
  arXiv  Detail & Related papers  (2019-11-07T13:48:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.