Blog: Survey of Optimizers

• URL: http://arxiv.org/abs/2608.28557v1
• Date: Fri, 28 Aug 2026 17:35:11 GMT
• Title: Blog: Survey of Optimizers
• Abstract summary: Survey organizes recent and training optimization methods along four axes: temporal estimation, update geometry, horizon management, and representation and systems.<n>It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficients, schedule-free training, small-batch corrections, and quantized states.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.

Related papers

• Reducing Per-Sample Harm in Stochastic Optimization [1.5229257192293202]
  Moderns combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.<n>We term this effect harm and formalize the parameter update as an optimization problem.<n>Because the exact formulation is intractable, we introduce a highly efficient proxy.
  arXiv  Detail & Related papers  (2026-06-28T18:01:27Z)
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms [3.5975968496682484]
  Moderns like Muon impose matrix-wise geometry constraints on their updates.<n>All current methods impose fixed LMO geometries for the update rules, chosen by-design or empirically, which are not optimal according to the problem's geometry.<n>We introduce a novel efficient datadriven criterion for dynamically choosing proxy-optimal update LMO geometries.
  arXiv  Detail & Related papers  (2026-05-19T12:47:41Z)
• Learn2Splat: Extending the Horizon of Learned 3DGS Optimization [53.913063313338874]
  We introduce a learned for 3DGS that avoids degradation over extended horizons without auxiliary mechanisms.<n>Results improved novel view quality while remaining stable over long horizons, with zero-shot generalization to unseen reconstruction settings.
  arXiv  Detail & Related papers  (2026-05-15T09:21:29Z)
• Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers [0.0]
  Training large language models requires optimization algorithms that are computationally and memory efficient at extreme scale.<n>This survey reviews design for large language models through a systems-and-optimization lens.
  arXiv  Detail & Related papers  (2026-05-09T21:34:28Z)
• Two-Stage Optimizer-Aware Online Data Selection for Large Language Models [49.576993784867035]
  We propose a principled-aware framework for gradient-based online data selection and reweighting in large language model fine-tuning.<n>Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the state.<n> Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.
  arXiv  Detail & Related papers  (2026-03-08T21:46:16Z)
• Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization [54.723834588133165]
  We study online linear optimization with matrix variables by the operatorAML, a setting where the geometry renders designing datadependent and efficient adaptive algorithms challenging.<n>We instantiate this framework with two efficient methods that avoid projections.<n>We show both methods admit closed-form updates match one-sided Shampoo's regret up to a constant factor, while significantly reducing computational cost.
  arXiv  Detail & Related papers  (2026-02-09T03:09:47Z)
• Towards Robust Scaling Laws for Optimizers [89.21160945066737]
  Empirical scaling laws are widely used to predict loss as model size and training data grow.<n>We show that Chinchilla-style scaling laws emerge naturally as a result of loss decomposition into irreducible, approximation, and optimization errors.
  arXiv  Detail & Related papers  (2026-02-07T21:40:33Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Improving Adaptive Moment Optimization via Preconditioner Diagonalization [11.01832755213396]
  We show that our approach can substantially enhance the convergence speed of modern adaptives.<n>For large language models like LLaMA, we can achieve a speedup of 2x compared to the baseline Adam.
  arXiv  Detail & Related papers  (2025-02-11T11:48:04Z)
• Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach [51.76826149868971]
  Policy evaluation via Monte Carlo simulation is at the core of many MC Reinforcement Learning (RL) algorithms. We propose as a quality index a surrogate of the mean squared error of a return estimator that uses trajectories of different lengths. We present an adaptive algorithm called Robust and Iterative Data collection strategy Optimization (RIDO)
  arXiv  Detail & Related papers  (2024-10-17T11:47:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.