Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

• URL: http://arxiv.org/abs/2608.12925v1
• Date: Thu, 13 Aug 2026 08:04:38 GMT
• Title: Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization
• Abstract summary: We develop an $textbfA$DMM-$textbfI$nspired $textbfM$omentum (AIM) framework based on residual-penalty variable splitting.<n>AIM recovers the exponential moving average of a multiplier update from an ADMM-style multiplier update.<n>We propose $textbfA$ccelerated $textbfR$esidual (RADAR) to improve the update direction and momentum estimation.
• Score: 16.249904972534114
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum recursion, update geometry, and acceleration remain only partially understood. We develop an $\textbf{A}$DMM-$\textbf{I}$nspired $\textbf{M}$omentum (AIM) framework based on residual-penalty variable splitting, which interprets momentum as a multiplier-like correction driven by the splitting residual. AIM recovers the exponential moving average of gradients from an ADMM-style multiplier update and separates two mechanisms that are usually intertwined in practical optimizers: the residual penalty determines the update geometry, whereas the approximation of the objective-related subproblem determines the acceleration form. Building on AIM, we propose $\textbf{R}$elativistic $\textbf{A}$daptive gradient $\textbf{D}$escent with $\textbf{A}$ccelerated $\textbf{R}$esidual (RADAR), which combines relativistic adaptive geometry, decoupled residual correction, and second-order momentum filtering to improve the update direction and momentum estimation. We establish stochastic convergence through a variance-perturbed Lyapunov drift analysis. Experiments on supervised vision learning, language modeling, and reinforcement learning show that RADAR achieves consistent improvements over strong adaptive optimizer baselines.

Related papers

• Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation [31.594936994611043]
  On-policy distillation (OPD) has emerged as an effective approach for large language model post-training.<n>We propose $$OPD, which uses discounted temporal credit assignment to balance long-horizon supervision and optimization stability.<n>We develop a reward-compatible bounded mixing mechanism for $mathrmOPD$ that balances verifiable outcome feedback with the discounted OPD advantage.
  arXiv  Detail & Related papers  (2026-09-15T10:12:56Z)
• Distribution-Aware Robust Bilevel Optimization: Quantile-Guided Huber Updates in Two-Timescale Stochastic Approximation [16.106678611834102]
  Bilevel optimization (BLO) is fundamental to hierarchical decision-making but suffers from critical instability under heavy-tailed noise.<n>Existing techniques typically rely on magnitude checks, which fail to distinguish informative geometric signals from outliers.
  arXiv  Detail & Related papers  (2026-06-21T10:57:37Z)
• Dimensionality Reduction for Robust Federated Learning: A Theoretical Analysis and Convergence Guarantee [14.954293251332894]
  Federated Learning (FL) enables clients to collaboratively train models without sharing raw data, but it is highly vulnerable to Byzantine attacks.<n>Existing robust approaches can neutralize these threats but incur substantial computational overhead.<n>We propose a universal acceleration framework for vector-level distance-based robust aggregators.
  arXiv  Detail & Related papers  (2026-05-27T11:39:47Z)
• Momentum Stability and Adaptive Control in Stochastic Reconfiguration [6.526210062544706]
  Variational Monte Carlo (VMC) combined with expressive neural network wavefunctions has become a powerful route to ground-state calculations.<n>We study efficient and stable wavefunction optimization via high-accuracy divergence via adaptive reconfiguration (SR)<n>We propose PRIME-SR, a tuning-free momentum-adaptive SR method based on effective spectral dimension and subspace overlap.
  arXiv  Detail & Related papers  (2026-04-20T14:49:30Z)
• GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control [0.0]
  We introduce GIRL (Generative Imagination Reinforcement Learning), a latent world-model framework that addresses this failure mode with two key components.<n> GIRL reduces latent rollout drift by 38 to 61 percent across tasks relative to DreamerV3, improves return, and requires fewer environment interactions on long-horizon tasks.<n>A distilled-prior variant reduces overhead and improves computational efficiency relative to the full model.
  arXiv  Detail & Related papers  (2026-04-08T17:14:21Z)
• Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations [55.047454145941366]
  Streaming Merging is an innovative model updating paradigm that conceptualizes merging as an iterative optimization process.<n> ARM is a strategy designed to approximate gradient descent dynamics.<n> ARM requires only early SFT checkpoints and, through iterative merging, surpasses the fully converged SFT model.
  arXiv  Detail & Related papers  (2026-02-03T08:15:57Z)
• A Simplified Analysis of SGD for Linear Regression with Weight Averaging [64.2393952273612]
  Recent work bycitetzou 2021benign provides sharp rates for SGD optimization in linear regression using constant learning rate.<n>We provide a simplified analysis recovering the same bias and variance bounds provided incitepzou 2021benign based on simple linear algebra tools.<n>We believe our work makes the analysis of gradient descent on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling.
  arXiv  Detail & Related papers  (2025-06-18T15:10:38Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Refining Adaptive Zeroth-Order Optimization at Ease [24.327161891577727]
  This paper introduces Refined Adaptive Zeroth-Order Optimization (R-AdaZO)<n>We first show the untapped variance reduction effect of first moment estimate on ZO gradient estimation.<n>We then refine the second moment estimate based on these variance-reduced gradient estimates to better capture the geometry of the optimization landscape.
  arXiv  Detail & Related papers  (2025-02-03T03:10:44Z)
• On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization [16.40968330148623]
  This paper proposes the $nu$PI algorithm and contributes an optimization perspective on Lagrange multiplier updates based on PI controllers. We provide theoretical and empirical insights explaining the inability of momentum methods to address the shortcomings of gradient descent-ascent. We prove that $nu$PI generalizes popular momentum methods for single-objective minimization.
  arXiv  Detail & Related papers  (2024-06-07T00:13:31Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization [80.8226518642952]
  We study optimization of areas under precision-recall curves (AUPRC), which is widely used for imbalanced tasks. We develop novel momentum methods with a better iteration of $O (1/epsilon4)$ for finding an $epsilon$stationary solution. We also design a novel family of adaptive methods with the same complexity of $O (1/epsilon4)$, which enjoy faster convergence in practice.
  arXiv  Detail & Related papers  (2021-07-02T16:21:52Z)
• Momentum Improves Normalized SGD [51.27183254738711]
  We show that adding momentum provably removes the need for large batch sizes on objectives. We show that our method is effective when employed on popular large scale tasks such as ResNet-50 and BERT pretraining.
  arXiv  Detail & Related papers  (2020-02-09T07:00:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.