SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

• URL: http://arxiv.org/abs/2505.09572v1
• Date: Wed, 14 May 2025 17:15:11 GMT
• Title: SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures
• Authors: Julian Kranz, Davide Gallon, Steffen Dereich, Arnulf Jentzen,
• Abstract summary: We study gradient flows for loss landscapes of fully connected feed forward neural networks with commonly used continuously differentiable activation functions.<n>We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an critical value.
• Score: 3.3123773366516645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study gradient flows for loss landscapes of fully connected feed forward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value. Moreover, we prove the existence of a threshold $\varepsilon>0$ such that the loss value of any gradient flow initialized at most $\varepsilon$ above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to real-world scenarios, where we observe an analogous behavior.

Related papers

• A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
  This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-18T16:46:08Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• A convergence result of a continuous model of deep learning via Łojasiewicz--Simon inequality [0.0]
  This focuses on a Wasserstein-type flow, which represents a process process of a continuous model of a Deep Neural Network (DNN) First, we establish the existence ar for an average loss of the model underL2regularization. We show existence of slope of of the loss as time as time as time of flow- optimization.
  arXiv  Detail & Related papers  (2023-11-26T17:44:29Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks [53.95175206863992]
  We study the type of solutions to which gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. We prove that although shallow ReLU networks are universal approximators, stable shallow networks are not.
  arXiv  Detail & Related papers  (2023-06-30T09:17:39Z)
• Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks [7.090165638014331]
  We prove linear convergence of gradient descent to a global minimum for the training of deep residual networks with constant layer width and smooth activation function. We show that the trained weights, as a function of the layer index, admits a scaling limit which is H"older continuous as the depth of the network tends to infinity.
  arXiv  Detail & Related papers  (2022-04-14T22:50:28Z)
• On Convergence of Training Loss Without Reaching Stationary Points [62.41370821014218]
  We show that Neural Network weight variables do not converge to stationary points where the gradient the loss function vanishes. We propose a new perspective based on ergodic theory dynamical systems.
  arXiv  Detail & Related papers  (2021-10-12T18:12:23Z)
• Continuous vs. Discrete Optimization of Deep Neural Networks [15.508460240818575]
  We show that over deep neural networks with homogeneous activations, gradient flow trajectories enjoy favorable curvature. This finding allows us to translate an analysis of gradient flow over deep linear neural networks into a guarantee that gradient descent efficiently converges to global minimum. We hypothesize that the theory of gradient flows will be central to unraveling mysteries behind deep learning.
  arXiv  Detail & Related papers  (2021-07-14T10:59:57Z)
• When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations? [51.1848572349154]
  We establish conditions under which gradient descent applied to fixed-width deep networks drives the logistic loss to zero. Our analysis applies for smoothed approximations to the ReLU, such as Swish and the Huberized ReLU.
  arXiv  Detail & Related papers  (2021-02-09T18:04:37Z)
• Asymptotic convergence rate of Dropout on shallow linear neural networks [0.0]
  We analyze the convergence on objective functions induced by Dropout and Dropconnect, when applying them to shallow linear Neural Networks. We obtain a local convergence proof of the gradient flow and a bound on the rate that depends on the data, the rate probability, and the width of the NN.
  arXiv  Detail & Related papers  (2020-12-01T19:02:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.