FlowAdam: Implicit Regularization via Geometry-Aware Soft Momentum Injection

• URL: http://arxiv.org/abs/2604.06652v1
• Date: Wed, 08 Apr 2026 03:57:52 GMT
• Title: FlowAdam: Implicit Regularization via Geometry-Aware Soft Momentum Injection
• Abstract summary: FlowAdam augments Adam with continuous gradient-flow integration via an ordinary differential equation (ODE)<n>Soft Momentum Injection blends ODE velocity with Adam's momentum during mode transitions.<n>Across coupled optimization benchmarks, the ODE integration provides implicit regularization, reducing held-out error by 10-22%.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adaptive moment methods such as Adam use a diagonal, coordinate-wise preconditioner based on exponential moving averages of squared gradients. This diagonal scaling is coordinate-system dependent and can struggle with dense or rotated parameter couplings, including those in matrix factorization, tensor decomposition, and graph neural networks, because it treats each parameter independently. We introduce FlowAdam, a hybrid optimizer that augments Adam with continuous gradient-flow integration via an ordinary differential equation (ODE). When EMA-based statistics detect landscape difficulty, FlowAdam switches to clipped ODE integration. Our central contribution is Soft Momentum Injection, which blends ODE velocity with Adam's momentum during mode transitions. This prevents the training collapse observed with naive hybrid approaches. Across coupled optimization benchmarks, the ODE integration provides implicit regularization, reducing held-out error by 10-22% on low-rank matrix/tensor recovery and 6% on Jester (real-world collaborative filtering), also surpassing tuned Lion and AdaBelief, while matching Adam on well-conditioned workloads (CIFAR-10). MovieLens-100K confirms benefits arise specifically from coupled parameter interactions rather than bias estimation. Ablation studies show that soft injection is essential, as hard replacement reduces accuracy from 100% to 82.5%.

Related papers

• Accelerating Diffusion Transformers with Gaussian Process Rectified Feature Cache [17.39547918764814]
  Diffusion Transformers have become the dominant paradigm in generative AI, but their high computational costs hinder real-time applications.<n>Prediction-based feature caching is widely used to accelerate diffusion transformers; however, as the number of steps increases, the deviation between its predictions and the reference full-compute trajectory gradually grows.<n>An intuitive idea is to use an online regression model to dynamically correct this deviation, but it faces the issue of label data being unavailable during the acceleration process.<n>This paper presents a statistical observation that the residuals between the features of full computation steps using caching methods and reference full-compute trajectory locally exhibit a
  arXiv  Detail & Related papers  (2026-09-05T08:54:54Z)
• What Does a Discrete Diffusion Model Learn? [71.03603607324338]
  discrete diffusion model learns: a denoiser, a score ratio, or a bridge plug-in predictor.<n>We start with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process.<n>Every identity is verified numerically, without approximation, on an exactly solvable model.
  arXiv  Detail & Related papers  (2026-07-06T17:56:11Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization [8.412194372218895]
  We introduce Singularity-aware Adam (S-Adam), a novel that stabilizes training by dynamically modulating step sizes based on local geometric instability.<n>S-Adam incorporates an adaptive damping mechanism that decelerates updates in high-instability regions while preserving fast convergence in smooth basins.<n> Empirical evaluations on optimal Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD.
  arXiv  Detail & Related papers  (2026-05-28T08:00:40Z)
• Directional Consistency as a Complementary Optimization Signal: The GONO Framework [0.0]
  We formalize an underexplored phenomenon in deep learning optimization: directional alignment and loss convergence can be decoupled.<n>We introduce GONO, which adapts Adam's momentum beta based on cc_t: amplifying momentum under directional consistency and suppressing it during oscillation.<n>We prove GONO matches Adam's convergence rate and reduces exactly to Adam when the signal is uninformative.
  arXiv  Detail & Related papers  (2026-05-07T17:05:05Z)
• AE-ViT: Stable Long-Horizon Parametric Partial Differential Equations Modeling [0.0]
  We propose a joint model consisting of a convolutional encoder, a transformer operating on latent representations, and a decoder for reconstruction.<n>Our approach combines the efficiency of latent evolution with the fidelity of full-field models, outperforming DL-ROMs, latent transformers, and plain ViTs in multi-field prediction.
  arXiv  Detail & Related papers  (2026-04-07T21:19:45Z)
• Decoupled Orthogonal Dynamics: Regularization for Deep Network Optimizers [7.170491058218013]
  We argue that magnitude and direction play distinct roles and should be decoupled in dynamics.<n>We propose Orthogonal Dynamics Decoupling and instantiate it as AdamO: an SGD-style update handles the onedimensional norm control, while Adam's adaptive preconditioning is confined to the tangential subspace.<n> Experiments on vision and language tasks show that AdamO improves generalization and stability over AdamW without introducing additional complex constraints.
  arXiv  Detail & Related papers  (2026-02-04T23:41:51Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• SPIRE: Conditional Personalization for Federated Diffusion Generative Models [7.8583640700306585]
  Shared Backbone Personal Identity Representation Embeddings (SPIRE) is a framework that casts per client diffusion based generation as conditional generation in FL.<n>SPIRE factorizes the network into (i) a high capacity global backbone that learns a population level score function and (ii) lightweight, learnable client embeddings that encode local data statistics.<n>Our analysis also hints at how client embeddings act as biases that steer a shared score network toward personalized distributions.
  arXiv  Detail & Related papers  (2025-06-14T01:40:31Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization [90.08459757321405]
  Federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead.<n>We propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates local model parameters and moment estimates.<n>By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error.
  arXiv  Detail & Related papers  (2024-05-28T07:56:49Z)
• Stable Anderson Acceleration for Deep Learning [0.0]
  Anderson acceleration (AA) is designed to speed-up fixed-point iterations like those arising from the iterative training of DL models. We combine AA with an adaptive moving average procedure that smoothes the oscillations and results in a more regular sequence of descent updates.
  arXiv  Detail & Related papers  (2021-10-26T14:52:16Z)
• Tom: Leveraging trend of the observed gradients for faster convergence [0.0]
  Tom is a novel variant of Adam that takes into account the trend observed for the gradients in the landscape in the loss traversed by the neural network. Tom outperforms Adagrad, Adadelta, RMSProp and Adam in terms of both accuracy and has a faster convergence.
  arXiv  Detail & Related papers  (2021-09-07T20:19:40Z)
• Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks [82.61182037130405]
  gradient descent (SGD) is the main approach for training deep networks. In this work, we compare Adam based variants based on the difference between the present and the past gradients. We have tested ensemble of networks and the fusion with ResNet50 trained with gradient descent.
  arXiv  Detail & Related papers  (2021-03-26T18:55:08Z)
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients [112.00379151834242]
  We propose adaptive learning rate principle, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance each coordinate. This results in faster adaptation, which leads more desirable empirical convergence behaviors.
  arXiv  Detail & Related papers  (2020-06-21T21:47:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.