On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer

• URL: http://arxiv.org/abs/2603.09952v1
• Date: Tue, 10 Mar 2026 17:49:19 GMT
• Title: On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer
• Abstract summary: We present a family of mean-normalized operator norms that admit layerwisesability and yield width-independent smoothness bounds.<n>We also show that textrmMuon can suffer an $mathcalO(qrtw) worst-case growth in the smoothness constant, whereas a new family of row-normalized to qmean$ achieves width-independent bounds.
• Score: 10.976013033990448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central question in modern deep learning is how to design optimizers whose behavior remains stable as the network width $w$ increases. We address this question by interpreting several widely used neural-network optimizers, including \textrm{AdamW} and \textrm{Muon}, as instances of steepest descent under matrix operator norms. This perspective links optimizer geometry with the Lipschitz structure of the network forward map, and enables width-independent control of both Lipschitz and smoothness constants. However, steepest-descent rules induced by standard $p \to q$ operator norms lack layerwise composability and therefore cannot provide width-independent bounds in deep architectures. We overcome this limitation by introducing a family of mean-normalized operator norms, denoted $\pmean \to \qmean$, that admit layerwise composability, yield width-independent smoothness bounds, and give rise to practical optimizers such as \emph{rescaled} \textrm{AdamW}, row normalization, and column normalization. The resulting learning rate width-aware scaling rules recover $μ$P scaling~\cite{yang2021tensor} as a special case and provide a principled mechanism for cross-width learning-rate transfer across a broad class of optimizers. We further show that \textrm{Muon} can suffer an $\mathcal{O}(\sqrt{w})$ worst-case growth in the smoothness constant, whereas a new family of row-normalized optimizers we propose achieves width-independent smoothness guarantees. Based on the observations, we propose MOGA (Matrix Operator Geometry Aware), a width-aware optimizer based only on row/column-wise normalization that enables stable learning-rate transfer across model widths. Large-scale pre-training on GPT-2 and LLaMA shows that MOGA, especially with row normalization, is competitive with Muon while being notably faster in large-token and low-loss regimes.

Related papers

• Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing [0.0]
  GMA replaces explicit pairwise query--key comparison with routing through $K$ learned Gaussian mixture components.<n>We formulate bidirectional and causal variants of GMA, provide an end-to-end differentiable parameterization of the Gaussian mixture components.<n>We analyze its responsibility-modulated structure, constrained non-negative low-rank affinity interpretation, and local routing stability.
  arXiv  Detail & Related papers  (2026-06-09T23:35:17Z)
• Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise [12.977441534320041]
  We show that a scale-invariant first-order method with spectral norm requires $(minm, n-frac3p-2p-1) calls.<n>We prove that a batched method achieves the matching bound of $(minm, n-frac5p2p-2p-2)$ when the norm is spectral and the Hessian is Lipschitz.
  arXiv  Detail & Related papers  (2026-05-18T15:13:18Z)
• Optimizer-Induced Mode Connectivity: From AdamW to Muon [50.258866851760416]
  We show that solutions from a single -- AdamW, Muon, or others in the Lion-$mathcalK$ family -- form a connected set at sufficiently large width.<n>We then characterize how-induced regions interact: at large width two different regions can be disjoint or overlap depending on regularization.
  arXiv  Detail & Related papers  (2026-05-11T05:07:36Z)
• Rethinking Language Model Scaling under Transferable Hypersphere Optimization [67.38433364607897]
  We introduce HyperP, the first framework for transferring optimal learning rates across model width, depth, training tokens, and expert-of-Experts (MoE)<n>A single base rate tuned data transfers across all compute budgets under HyperP, yielding $1.58times over a strong Muon baseline at $6times1021$ FLOPs.<n>We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities.
  arXiv  Detail & Related papers  (2026-03-30T17:51:47Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Singular Bayesian Neural Networks [1.2891210250935148]
  Bayesian neural networks promise calibrated uncertainty but require $O(mn)$ parameters for standard mean-field Gaussian posteriors.<n>We induce a posterior that is singular with respect to the Lebesgue measure, concentrating on the rank-$r$ manifold.<n>We derive PAC-Bayes generalization bounds whose complexity term scales as $sqrtr(m+n)$ instead of $sqrtm n$, and prove loss bounds that decompose the error into optimization and rank-induced bias.
  arXiv  Detail & Related papers  (2026-01-30T23:06:34Z)
• Manifold constrained steepest descent [0.0]
  We propose emphManifold Constrained Steepest Descent (MCSD), a single-loop framework for optimization over manifold.<n>We also introduce emphSPEL, the spectral specialization of MCSD on the Stiefel manifold.
  arXiv  Detail & Related papers  (2026-01-29T10:08:37Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Minimax Adaptive Online Nonparametric Regression over Besov Spaces [8.691764858459623]
  We study online adversarial regression with convex losses against a rich class of continuous yet highly irregular prediction rules.<n>We introduce an adaptive wavelet-based algorithm that performs sequential prediction without prior knowledge of $(s,p,q)$.<n>We also design a locally adaptive extension capable of dynamically tracking spatially inhomogeneous smoothness.
  arXiv  Detail & Related papers  (2025-05-26T09:23:11Z)
• Position: Curvature Matrices Should Be Democratized via Linear Operators [6.946287154076936]
  Linear operators provide a general, scalable, and user-friendly abstraction to handle curvature matrices.<n>$textitcurvlinops$ is a library that provides curvature matrices through a unified linear operator interface.<n>We demonstrate with $textitcurvlinops$ how this interface can hide complexity, simplify applications, be interoperable with other libraries, and scale to large NNs.
  arXiv  Detail & Related papers  (2025-01-31T14:46:30Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Distributed Extra-gradient with Optimal Complexity and Communication Guarantees [60.571030754252824]
  We consider monotone variational inequality (VI) problems in multi-GPU settings where multiple processors/workers/clients have access to local dual vectors. Extra-gradient, which is a de facto algorithm for monotone VI problems, has not been designed to be communication-efficient. We propose a quantized generalized extra-gradient (Q-GenX), which is an unbiased and adaptive compression method tailored to solve VIs.
  arXiv  Detail & Related papers  (2023-08-17T21:15:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.