PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$-Norm Steepest Descent

• URL: http://arxiv.org/abs/2605.10335v1
• Date: Mon, 11 May 2026 10:36:27 GMT
• Title: PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$-Norm Steepest Descent
• Abstract summary: Adaptives have become the default standard for training large-scale neural networks such as Transformers.<n>We introduce PowerStep, a memory-efficient that achieves a momentum-wise adaptivity without second-moment statistics.<n>Experiments on Transformer models demonstrate that PowerStep matches Adams convergence speed while halving memory.
• Score: 29.903629736539013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adaptive optimizers, most notably Adam, have become the default standard for training large-scale neural networks such as Transformers. These methods maintain running estimates of gradient first and second moments, incurring substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by steepest descent under an $\ell_p$-norm geometry, we show that applying a nonlinear transform directly to a momentum buffer yields coordinate-wise adaptivity. We prove that PowerStep converges at the optimal $O(1/\sqrt{T})$ rate for non-convex stochastic optimization. Extensive experiments on Transformer models ranging from 124M to 235B parameters demonstrate that PowerStep matches Adam's convergence speed while halving optimizer memory. Furthermore, when combined with aggressive \texttt{int8} quantization, PowerStep remains numerically stable and reduces optimizer memory by $\sim\!8\times$ compared to full-precision Adam. PowerStep thus provides a principled, scalable and resource-efficient alternative for large-scale training. Code is available at https://github.com/yaolubrain/PowerStep.

Related papers

• Second-Moment Memory in Coordinatewise Adam [0.0]
  We show that second-moment memory can itself suppress progress toward optimum gradients.<n>Long second-moment memory can slow optimization even when the gradient noise has finite variance.
  arXiv  Detail & Related papers  (2026-08-16T15:56:36Z)
• Gefen: Optimized Stochastic Optimizer [32.771995350910125]
  AdamW is a default for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory.<n>We propose Gefen, a memory-efficient codebook that shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook.<n>We show that Gefen reduces AdamW's memory footprint by 8x while maintaining the same performance, corresponding to a reduction of 6.5 GiB per billion parameters.
  arXiv  Detail & Related papers  (2026-06-11T20:38:09Z)
• Rethinking Language Model Scaling under Transferable Hypersphere Optimization [67.38433364607897]
  We introduce HyperP, the first framework for transferring optimal learning rates across model width, depth, training tokens, and expert-of-Experts (MoE)<n>A single base rate tuned data transfers across all compute budgets under HyperP, yielding $1.58times over a strong Muon baseline at $6times1021$ FLOPs.<n>We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities.
  arXiv  Detail & Related papers  (2026-03-30T17:51:47Z)
• Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergence [5.683382109687174]
  Memory-efficient optimization methods have recently gained increasing attention for scaling full-tuning of of large language models under the GPU-memory bottleneck.<n>We propose Omni-O-Masked Descent (OMGD) as an approach to memory-efficient training.
  arXiv  Detail & Related papers  (2026-03-06T06:47:52Z)
• Progressive Power Homotopy for Non-convex Optimization [5.737648067191245]
  We propose a novel first-order method for non- optimization of the form $max_bmwinmathbbdmathbbdmathR bmxsimmathcalD.<n>We show that Prog-PowerHP converges to a small neighborhood of the global optimum with mild complexity scaling nearly as $O(d2varepsilon-2)$.
  arXiv  Detail & Related papers  (2026-01-22T12:44:25Z)
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales [55.91454326946738]
  We study how the optimal learning rate and weight decay should scale with model width and depth for a wide range of languages.<n>We find that scaling the learning rate according to $$P improves transfer, but can still suffer from significant finite-width deviations.<n>For compute-optimal scaling, we find scaling independent weight decay as $1/mathrmwidth$ is nearly optimal across languages.
  arXiv  Detail & Related papers  (2025-12-05T11:03:41Z)
• FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models [49.397861654088636]
  We propose a two-step procedure to approximate SVD/QR-based gradient projections into lower-dimensional spaces.<n>We show that our strategy achieves faster runtime and reduced memory usage by up to $25%$ across different model sizes.
  arXiv  Detail & Related papers  (2025-05-23T14:37:00Z)
• VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergence [6.574641780732972]
  Large-scale non problems are common in deep learning.<n>First-order (FO)s serve as today's baselines.<n>ZO algorithms reduce the size of computation and memory costs.<n>VAMO achieves these gains with smaller dynamic memory requirements.
  arXiv  Detail & Related papers  (2025-05-20T05:31:15Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.