Adaptive Momentum and Nonlinear Damping for Neural Network Training

• URL: http://arxiv.org/abs/2602.00334v1
• Date: Fri, 30 Jan 2026 21:35:55 GMT
• Title: Adaptive Momentum and Nonlinear Damping for Neural Network Training
• Abstract summary: We propose a continuous-time scheme for large-scale optimization that introduces individual, adaptive momentum coefficients regulated by the kinetic energy of each model parameter.<n>We demonstrate that our adaptive friction can be related to cubic damping, a suppression mechanism from structural dynamics.
• Score: 1.0149624140985478
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a continuous-time scheme for large-scale optimization that introduces individual, adaptive momentum coefficients regulated by the kinetic energy of each model parameter. This approach automatically adjusts to local landscape curvature to maintain stability without sacrificing convergence speed. We demonstrate that our adaptive friction can be related to cubic damping, a suppression mechanism from structural dynamics. Furthermore, we introduce two specific optimization schemes by augmenting the continuous dynamics of mSGD and Adam with a cubic damping term. Empirically, our methods demonstrate robustness and match or outperform Adam on training ViT, BERT, and GPT2 tasks where mSGD typically struggles. We further provide theoretical results establishing the exponential convergence of the proposed schemes.

Related papers

• When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay [6.625270757109195]
  We show that a single quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect.<n>This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes.
  arXiv  Detail & Related papers  (2026-09-08T17:46:17Z)
• Adaptive Equilibrium: Dynamic Weighting Framework for Generalized Interruption of DeepFake Models [24.55206893241103]
  We argue that achieving high and uniform effectiveness requires resolving this imbalance by reaching an adaptive equilibrium.<n>We propose the Adaptive Equilibrium Framework (AEF), which employs a dynamic weighting mechanism that utilizes real-time loss feedback to adaptively assign greater interruption weights to the most resistant models.
  arXiv  Detail & Related papers  (2026-05-01T06:19:59Z)
• Momentum-constrained Hybrid Heuristic Trajectory Optimization Framework with Residual-enhanced DRL for Visually Impaired Scenarios [20.048701695146963]
  This paper proposes a Momentum-Constrained Hybrid Heuristic Trajectory Optimization Framework (MHHTOF)<n>To balance multiple objectives of comfort and safety, the framework designs a Heuristic Trajectory Sampling Cluster (HTSC) with a Momentum-Constrained Trajectory Optimization (MTO)<n>In complex dynamic scenarios, MHHTOF demonstrates stable velocity and acceleration curves with reduced risk.
  arXiv  Detail & Related papers  (2026-04-16T13:16:00Z)
• JAWS: Enhancing Long-term Rollout of Neural Operators via Spatially-Adaptive Jacobian Regularization [0.0]
  Jacobian-Adaptive Weighting for Stability (JAWS) is a probabilistic regularization strategy designed to mitigate these limitations.<n>By framing operator learning as Maximum A Posteriori (MAP) estimation with spatially heteroscedastic uncertainty, JAWS dynamically modulates the regularization strength based on local physical complexity.<n> Experiments demonstrate that this spatially-adaptive prior serves as an effective spectral pre-conditioner, which reduces the base operator's burden of handling high-frequency instabilities.
  arXiv  Detail & Related papers  (2026-03-04T06:15:09Z)
• On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective [2.268525139011456]
  We study the convergence dynamics of Gradient Descent (GD) in a minimal binary classification setting.<n>We prove that while GD successfully converges to an optimal robustness margin, this convergence occurs at a prohibitively slow rate.<n>Our theoretical guarantees are derived via a rigorous analysis of the GD trajectories across the distinct activation patterns of the model.
  arXiv  Detail & Related papers  (2026-03-02T17:13:33Z)
• Hyperparameter Trajectory Inference with Conditional Lagrangian Optimal Transport [51.56484100374058]
  Post-deployment, user preferences can evolve, making initial settings undesirable.<n>We learn, from observed data, how a NN's conditional output distribution changes with its hyper parameters.<n>We construct a surrogate model that approximates the NN at unobserved hyper parameters.
  arXiv  Detail & Related papers  (2026-03-02T11:55:02Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inference [1.7523718031184992]
  We identify a fundamental mechanism for this failure: textbfPremature Mode Collapse.<n>We propose textbfEfficient Piecewise Hybrid Adaptive Stability Control (EPH-ASC), an adaptive scheduling algorithm that monitors the stability of the inference process.
  arXiv  Detail & Related papers  (2026-01-30T14:47:18Z)
• Towards a Unified Analysis of Neural Networks in Nonparametric Instrumental Variable Regression: Optimization and Generalization [66.08522228989634]
  We establish the first global convergence result of neural networks for two stage least squares (2SLS) approach in nonparametric instrumental variable regression (NPIV)<n>This is achieved by adopting a lifted perspective through mean-field Langevin dynamics (MFLD)
  arXiv  Detail & Related papers  (2025-11-18T17:51:17Z)
• A Physics-Inspired Optimizer: Velocity Regularized Adam [9.38448580878081]
  We introduce Velocity-Regularized Adamvolution (VRAdam) for training deep neural networks.<n>VRAdam adds a higher order penalty on the learning rate based on the velocity.<n>We demonstrate that VRAdam exceeds the performance against standard convexs including AdamW.
  arXiv  Detail & Related papers  (2025-05-19T14:51:40Z)
• Efficient Design of Compliant Mechanisms Using Multi-Objective Optimization [50.24983453990065]
  We address the synthesis of a compliant cross-hinge mechanism capable of large angular strokes.<n>We formulate a multi-objective optimization problem based on kinetostatic performance measures.
  arXiv  Detail & Related papers  (2025-04-23T06:29:10Z)
• Conformal Symplectic Optimization for Stable Reinforcement Learning [21.491621524500736]
  By utilizing relativistic kinetic energy, RAD incorporates from special relativity and limits parameter updates below a finite speed, effectively mitigating abnormal influences.<n> Notably, RAD achieves up to a 155.1% performance improvement, showcasing its efficacy in training Atari games.
  arXiv  Detail & Related papers  (2024-12-03T09:07:31Z)
• Neural Projected Quantum Dynamics: a systematic study [0.0]
  We investigate the challenge of classical simulation of unitary quantum dynamics with variational Monte Carlo approaches.<n>We provide a thorough formalization and significant improvements to the projected time-dependent Monte Carlo (p-tVMC) method.<n>This work establishes p-tVMC as a powerful framework for simulating the dynamics of large-scale two-dimensional quantum systems.
  arXiv  Detail & Related papers  (2024-10-14T17:01:33Z)
• Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy [75.15685966213832]
  We analyze the rich directional structure of optimization trajectories represented by their pointwise parameters. We show that training only scalar batchnorm parameters some while into training matches the performance of training the entire network.
  arXiv  Detail & Related papers  (2024-03-12T07:32:47Z)
• Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction [49.66486092259376]
  The mean-field Langevin dynamics (MFLD) is a nonlinear generalization of the Langevin dynamics that incorporates a distribution-dependent drift. Recent works have shown that MFLD globally minimizes an entropy-regularized convex functional in the space of measures. We provide a framework to prove a uniform-in-time propagation of chaos for MFLD that takes into account the errors due to finite-particle approximation, time-discretization, and gradient approximation.
  arXiv  Detail & Related papers  (2023-06-12T16:28:11Z)
• Nesterov Meets Optimism: Rate-Optimal Separable Minimax Optimization [108.35402316802765]
  We propose a new first-order optimization algorithm -- AcceleratedGradient-OptimisticGradient (AG-OG) Ascent. We show that AG-OG achieves the optimal convergence rate (up to a constant) for a variety of settings. We extend our algorithm to extend the setting and achieve the optimal convergence rate in both bi-SC-SC and bi-C-SC settings.
  arXiv  Detail & Related papers  (2022-10-31T17:59:29Z)
• Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
  convergence rate analysis of the mean field Langevin dynamics is presented. $p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
  arXiv  Detail & Related papers  (2022-01-25T17:13:56Z)
• Role of stochastic noise and generalization error in the time propagation of neural-network quantum states [0.0]
  Neural-network quantum states (NQS) have been shown to be a suitable variational ansatz to simulate out-of-equilibrium dynamics. We show that stable and accurate time propagation can be achieved in regimes of sufficiently regularized variational dynamics.
  arXiv  Detail & Related papers  (2021-05-03T17:55:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.