FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

• URL: http://arxiv.org/abs/2606.02365v1
• Date: Mon, 01 Jun 2026 15:13:28 GMT
• Title: FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
• Abstract summary: We provide a theoretical study of staleness through the complementary lenses of convergence and stability.<n>We propose an adaptive algorithm that stabilizes training by dynamically controlling both the damping factor and the eigendecomposition frequency.<n> Experimental results demonstrate that reduces wall-clock time compared to standard Shampoo while maintaining robust convergence.
• Score: 9.04365834292336
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Shampoo is attracting considerable attention for its superior performance on large-scale optimization benchmarks; yet it faces a significant practical bottleneck: the prohibitive computational overhead of matrix inversion. To mitigate this, practitioners typically rely on stale preconditioner updates, creating a fundamental trade-off between computational efficiency and optimization fidelity. In this work, we provide a theoretical study of staleness through the complementary lenses of convergence and stability. While staleness improves computational efficiency, it inherently degrades performance and introduces numerical instability. Crucially, we identify that damping, acting as a numerical stabilizer, can effectively suppress these negative effects. Guided by this analysis, we propose FOAM, an adaptive algorithm that stabilizes training by dynamically controlling both the damping factor and the eigendecomposition frequency based on an approximation of the staleness-oriented error. Experimental results demonstrate that FOAM reduces wall-clock time compared to standard Shampoo while maintaining robust convergence.

Related papers

• Taming Curvature: Architecture Warm-Up for Stable Transformer Training [52.6838951211596]
  We introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power with Hessian-vector products.<n>We find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth.<n> Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities.
  arXiv  Detail & Related papers  (2026-06-15T14:16:56Z)
• Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models [24.301295080069476]
  Looped Language Models (LoopLMs) enable efficient latent reasoning through depth recurrence, yet exhibit unreliable test-time scaling behavior.<n>We propose STARS (ST-driven Recurrent Scaling), a training framework that constrains latent states to approach reductionally stable fixed points.
  arXiv  Detail & Related papers  (2026-05-26T09:11:12Z)
• Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization [69.03519018775779]
  Group Relative Policy Optimization is essential for aligning video diffusion models with human preferences.<n>Existing efficiency methods compromise optimization, exhibiting severe instability and failing to reach full trajectory performance.<n>We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets.
  arXiv  Detail & Related papers  (2026-05-15T14:13:39Z)
• Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data [77.19117648492104]
  We present Federated Robust Curvature Optimization (FedRCO), a novel second-order optimization framework designed to improve convergence speed and reduce communication cost.<n>FedRCO addresses these challenges by integrating an efficient approximate curvature with a provable stability mechanism.<n>We show that FedRCO achieves superior robustness against diverse non-IID scenarios while achieving higher accuracy and faster convergence than both state-of-the-art first-order and second-order methods.
  arXiv  Detail & Related papers  (2026-03-30T11:37:46Z)
• JAWS: Enhancing Long-term Rollout of Neural Operators via Spatially-Adaptive Jacobian Regularization [0.0]
  Jacobian-Adaptive Weighting for Stability (JAWS) is a probabilistic regularization strategy designed to mitigate these limitations.<n>By framing operator learning as Maximum A Posteriori (MAP) estimation with spatially heteroscedastic uncertainty, JAWS dynamically modulates the regularization strength based on local physical complexity.<n> Experiments demonstrate that this spatially-adaptive prior serves as an effective spectral pre-conditioner, which reduces the base operator's burden of handling high-frequency instabilities.
  arXiv  Detail & Related papers  (2026-03-04T06:15:09Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learning [10.698225972251839]
  Zeroth-order (ZO) optimization has gained attention as a memory-efficient alternative to first-order (FO) methods.<n>We show that ZO optimization naturally leads to flatter loss landscapes, which in turn reduce forgetting in continuous learning.<n>This stability comes at a cost of plasticity: due to its imprecise gradient estimates and slower convergence, ZO optimization tends to be less effective than FO in acquiring new task-specific knowledge.<n>We propose ZO-FC, a simple but effective approach that applies ZO optimization to a single adapter-based PEFT module with FO optimized classifier.
  arXiv  Detail & Related papers  (2025-10-23T21:54:00Z)
• Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner [25.56447816096372]
  Recent success of Shampoo in the computationPerf contest has sparked renewed interest in Kroneckerfactorization-based algorithms for training neural networks.<n>We show that grafting from Adam directly mitigates the staleness and mis-scaling of the preconditioner's eigenvalues.<n>We propose an adaptive criterion for determining the eigenbasis frequency motivated by terminating a warm-started QR algorithm.
  arXiv  Detail & Related papers  (2025-06-04T05:55:41Z)
• Fast and Accurate Power Load Data Completion via Regularization-optimized Low-Rank Factorization [10.713082490316111]
  Low-rank representation learning has emerged as a powerful tool for recovering missing values in power load data.<n>Regularizationtemporal Low-Rank Factorization model is favoured for its efficiency and interpretability.<n>We propose a Regularizationtemporal Low-Rank Factorization controller, which adapts to adjust the regularization coefficient.
  arXiv  Detail & Related papers  (2025-05-25T13:07:55Z)
• On the Power of Perturbation under Sampling in Solving Extensive-Form Games [56.013335390600524]
  We investigate how perturbation does and does not improve the Follow-the-Regularized-Leader (FTRL) algorithm in solving extensive-form games under sampling.<n>We present a unified framework for textitPerturbed FTRL algorithms and study two variants: PFTRL-KL and PFTRL-RKL.
  arXiv  Detail & Related papers  (2025-01-28T00:29:38Z)
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality [131.45028999325797]
  We develop a doubly robust off-policy AC (DR-Off-PAC) for discounted MDP. DR-Off-PAC adopts a single timescale structure, in which both actor and critics are updated simultaneously with constant stepsize. We study the finite-time convergence rate and characterize the sample complexity for DR-Off-PAC to attain an $epsilon$-accurate optimal policy.
  arXiv  Detail & Related papers  (2021-02-23T18:56:13Z)
• Distributionally Robust Bayesian Optimization [121.71766171427433]
  We present a novel distributionally robust Bayesian optimization algorithm (DRBO) for zeroth-order, noisy optimization. Our algorithm provably obtains sub-linear robust regret in various settings. We demonstrate the robust performance of our method on both synthetic and real-world benchmarks.
  arXiv  Detail & Related papers  (2020-02-20T22:04:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.