Understanding Optimization in Deep Learning with Central Flows

• URL: http://arxiv.org/abs/2410.24206v2
• Date: Thu, 25 Sep 2025 14:29:29 GMT
• Title: Understanding Optimization in Deep Learning with Central Flows
• Authors: Jeremy M. Cohen, Alex Damian, Ameet Talwalkar, J. Zico Kolter, Jason D. Lee,
• Abstract summary: We develop theory that can describe the dynamics of optimization in a complex regime.<n>Our results suggest that central flows can be a valuable theoretical tool for reasoning about optimization in deep learning.
• Score: 95.5647720254338
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional theories of optimization cannot describe the dynamics of optimization in deep learning, even in the simple setting of deterministic training. The challenge is that optimizers typically operate in a complex, oscillatory regime called the "edge of stability." In this paper, we develop theory that can describe the dynamics of optimization in this regime. Our key insight is that while the *exact* trajectory of an oscillatory optimizer may be challenging to analyze, the *time-averaged* (i.e. smoothed) trajectory is often much more tractable. To analyze an optimizer, we derive a differential equation called a "central flow" that characterizes this time-averaged trajectory. We empirically show that these central flows can predict long-term optimization trajectories for generic neural networks with a high degree of numerical accuracy. By interpreting these central flows, we are able to understand how gradient descent makes progress even as the loss sometimes goes up; how adaptive optimizers "adapt" to the local loss landscape; and how adaptive optimizers implicitly navigate towards regions where they can take larger steps. Our results suggest that central flows can be a valuable theoretical tool for reasoning about optimization in deep learning.

Related papers

• Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization [0.0]
  We use trajectory data to learn the continuous-time dynamics associated with gradient descent, Newton's method, and ADAM optimization.<n>The discovered gradient is then solved as a surrogate for the original optimization problem.<n>We demonstrate the efficacy of this approach on several standard problems from engineering mechanics and scientific machine learning.
  arXiv  Detail & Related papers  (2026-02-13T22:44:33Z)
• Optimizing Optimizers for Fast Gradient-Based Learning [53.81268610971847]
  We lay the theoretical foundation for automating design in gradient-based learning.<n>By treating gradient loss signals as a function that translates to parameter motions, the problem reduces to a family of convex optimization problems.
  arXiv  Detail & Related papers  (2025-12-06T09:50:41Z)
• Reinforcement Learning for Accelerated Aerodynamic Shape Optimisation [0.0]
  We introduce a reinforcement learning based adaptive optimization algorithm for aerodynamic shape optimization.<n>The goals are to minimize computational effort, and to use the observed optimization results for interpretation of the discovered extrema.
  arXiv  Detail & Related papers  (2025-07-23T09:14:25Z)
• Optimizers Qualitatively Alter Solutions And We Should Leverage This [62.662640460717476]
  Deep Neural Networks (DNNs) can not guarantee convergence to a unique global minimum of the loss when using only local information, such as SGD.<n>We argue that the community should aim at understanding the biases of already existing methods, as well as aim to build new DNNs with the explicit intent of inducing certain properties of the solution.
  arXiv  Detail & Related papers  (2025-07-16T13:33:31Z)
• Hindsight-Guided Momentum (HGM) Optimizer: An Approach to Adaptive Learning Rate [0.0]
  We introduce Hindsight-Guided Momentum, a first-order optimization algorithm that adaptively scales learning rates based on recent updates.<n>HGM addresses this by a hindsight mechanism that accelerates the learning rate between coherent and conflicting directions.
  arXiv  Detail & Related papers  (2025-06-22T08:02:19Z)
• Revisiting the Initial Steps in Adaptive Gradient Descent Optimization [6.468625143772815]
  Adaptive gradient optimization methods, such as Adam, are prevalent in training deep neural networks across diverse machine learning tasks. These methods often suffer from suboptimal generalization compared to descent gradient (SGD) and exhibit instability. We introduce simple yet effective solutions: initializing the second-order moment estimation with non-zero values.
  arXiv  Detail & Related papers  (2024-12-03T04:28:14Z)
• A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning [74.80956524812714]
  We tackle the general differentiable meta learning problem that is ubiquitous in modern deep learning. These problems are often formalized as Bi-Level optimizations (BLO) We introduce a novel perspective by turning a given BLO problem into a ii optimization, where the inner loss function becomes a smooth distribution, and the outer loss becomes an expected loss over the inner distribution.
  arXiv  Detail & Related papers  (2024-10-14T12:10:06Z)
• Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy [75.15685966213832]
  We analyze the rich directional structure of optimization trajectories represented by their pointwise parameters. We show that training only scalar batchnorm parameters some while into training matches the performance of training the entire network.
  arXiv  Detail & Related papers  (2024-03-12T07:32:47Z)
• Analyzing and Enhancing the Backward-Pass Convergence of Unrolled Optimization [50.38518771642365]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. A central challenge in this setting is backpropagation through the solution of an optimization problem, which often lacks a closed form. This paper provides theoretical insights into the backward pass of unrolled optimization, showing that it is equivalent to the solution of a linear system by a particular iterative method. A system called Folded Optimization is proposed to construct more efficient backpropagation rules from unrolled solver implementations.
  arXiv  Detail & Related papers  (2023-12-28T23:15:18Z)
• Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter $ε$ via Gradient Magnitude Histogram Analysis [0.7366405857677226]
  We introduce a new framework based on the empirical probability density function of the loss's magnitude, termed the "gradient magnitude histogram" We propose a novel algorithm using gradient magnitude histograms to automatically estimate a refined and accurate search space for the optimal safeguard.
  arXiv  Detail & Related papers  (2023-11-20T04:34:19Z)
• Understanding Optimization of Deep Learning via Jacobian Matrix and Lipschitz Constant [18.592094066642364]
  This article provides a comprehensive understanding of optimization in deep learning. We focus on the challenges of gradient vanishing and gradient exploding, which normally lead to diminished model representational ability and training instability, respectively. To help understand the current optimization methodologies, we categorize them into two classes: explicit optimization and implicit optimization.
  arXiv  Detail & Related papers  (2023-06-15T17:59:27Z)
• Break a Lag: Triple Exponential Moving Average for Enhanced Optimization [2.0199251985015434]
  We introduce Fast Adaptive Moment Estimation (FAME), a novel optimization technique that leverages the power of Triple Exponential Moving Average. FAME enhances responsiveness to data dynamics, mitigates trend identification lag, and optimize learning efficiency. Our comprehensive evaluation encompasses different computer vision tasks including image classification, object detection, and semantic segmentation, integrating FAME into 30 distinct architectures.
  arXiv  Detail & Related papers  (2023-06-02T10:29:33Z)
• Backpropagation of Unrolled Solvers with Folded Optimization [55.04219793298687]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation.
  arXiv  Detail & Related papers  (2023-01-28T01:50:42Z)
• Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence [31.59453616577858]
  We show that a distributed system can withstand noisier zeroth-order agents but can even benefit from such agents into the optimization process. Our results hold both convex and non-zero-th order optimization objectives while they could still contribute to joint optimization tasks.
  arXiv  Detail & Related papers  (2022-10-14T10:54:11Z)
• Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks [12.90962626557934]
  Conal neural networks (CNNs) have shown very appealing performance for many computer vision applications. In this paper, we propose a moment centralization-based SGD datasets for CNNs. The proposed moment centralization is generic in nature and can be integrated with any of the existing adaptive momentum-baseds.
  arXiv  Detail & Related papers  (2022-07-19T04:38:01Z)
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks [21.63353575405414]
  We study the implicit bias of adaptive optimization algorithms on homogeneous neural networks. It is the first work to study the convergent direction of adaptive optimizations on non-linear deep neural networks.
  arXiv  Detail & Related papers  (2020-12-11T11:15:32Z)
• Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering [53.523517926927894]
  We explore the use of exact per-sample Hessian-vector products and gradients to construct self-tuning quadratics. We prove that our model-based procedure converges in noisy gradient setting. This is an interesting step for constructing self-tuning quadratics.
  arXiv  Detail & Related papers  (2020-11-09T22:07:30Z)
• Fast Rates for Contextual Linear Optimization [52.39202699484225]
  We show that a naive plug-in approach achieves regret convergence rates that are significantly faster than methods that directly optimize downstream decision performance. Our results are overall positive for practice: predictive models are easy and fast to train using existing tools, simple to interpret, and, as we show, lead to decisions that perform very well.
  arXiv  Detail & Related papers  (2020-11-05T18:43:59Z)
• A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning [95.85269649177336]
  ZO optimization iteratively performs three major steps: gradient estimation, descent direction, and solution update. We demonstrate promising applications of ZO optimization, such as evaluating and generating explanations from black-box deep learning models, and efficient online sensor management.
  arXiv  Detail & Related papers  (2020-06-11T06:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.