Convergence of Spectral Descent for Non-smooth Optimization

• URL: http://arxiv.org/abs/2605.26977v1
• Date: Tue, 26 May 2026 13:02:49 GMT
• Title: Convergence of Spectral Descent for Non-smooth Optimization
• Abstract summary: Current convergence guarantees for Muon rely heavily on smoothness assumptions, leaving its non-smooth convergence behavior largely unexplored.<n>We establish global linear convergence for both SD and TSD in non-smooth convex formulations.<n>We apply our theoretical framework to robust low-rank matrix recovery under mixed sparse and dense noise regimes.
• Score: 6.918561684491688
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Muon optimizer has recently demonstrated remarkable empirical success in training large language models. However, the theoretical understanding of its mechanisms remains limited. Current convergence guarantees for Muon rely heavily on smoothness assumptions, leaving its non-smooth convergence behavior largely unexplored. In this work, we take a step toward bridging this gap by investigating Spectral Descent (SD), a simplified variant of Muon, together with its truncated counterpart, Truncated Spectral Descent (TSD). Under convexity, Lipschitz continuity, and sharpness conditions, we establish global linear convergence for both SD and TSD in non-smooth convex formulations. We also study regularized variants equipped with decoupled weight decay and derive sublinear convergence guarantees through their connection with Frank-Wolfe methods. Finally, we apply our theoretical framework to robust low-rank matrix recovery under mixed sparse and dense noise regimes and provide rigorous recovery guarantees. Numerical experiments support the theoretical findings and demonstrate the effectiveness of Muon-type methods for non-smooth optimization.

Related papers

• MiNCE: Nonparametric, Strongly Consistent Confidence Envelopes for Band-Limited Functions and their Smoothed Spectra [0.42970700836450487]
  We study a nonparametric approach to constructing nonasymptotic, simultaneous confidence regions for band-limited functions.<n>We establish the strong uniform consistency of the resulting bands, both for noise-free and noisy observation models.<n>We extend this formulation to the frequency domain, deriving nonasymptotic, simultaneous, strongly uniformly consistent confidence bands for the smoothed spectra.
  arXiv  Detail & Related papers  (2026-09-08T20:37:27Z)
• Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives [48.91820744387913]
  We develop preconditioned gradient methods with a focus on Muon and Scions.<n>We introduce a family of algorithms that can handle a wide variety of convex and non-reduced constraints.
  arXiv  Detail & Related papers  (2026-05-12T09:36:13Z)
• DOC-GS: Dual-Domain Observation and Calibration for Reliable Sparse-View Gaussian Splatting [80.43237927269575]
  In this paper, we revisit sparse-view 3DGS reconstruction from a new perspective.<n>We identify the core challenge as the unobservability of Gaussian primitive reliability.<n>Motivated by this observation, we propose a unified Dual-domain Observation and geometrics in rendered images framework.
  arXiv  Detail & Related papers  (2026-04-08T07:01:24Z)
• Preconditioning Benefits of Spectral Orthogonalization in Muon [50.62925024212989]
  We study the effectiveness of a simplified variant of Muon in two case studies: matrix factorization and in-context learning of linear transformers.<n>Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior.
  arXiv  Detail & Related papers  (2026-01-20T00:08:31Z)
• On the Convergence of Muon and Beyond [31.900178928104648]
  We provide the first proof that variance reduction enables Muon-MVR2 to attain the optimal complexity.<n>Overall, this work offers the first proof of optimality for a Muon-style.
  arXiv  Detail & Related papers  (2025-09-19T09:43:37Z)
• Revisiting Convergence: Shuffling Complexity Beyond Lipschitz Smoothness [50.78508362183774]
  Shuffling-type gradient methods are favored in practice for their simplicity and rapid empirical performance.<n>Most require the Lipschitz condition, which is often not met in common machine learning schemes.
  arXiv  Detail & Related papers  (2025-07-11T15:36:48Z)
• Convergence Bound and Critical Batch Size of Muon Optimizer [1.2289361708127877]
  We provide convergence proofs for Muon across four practical settings.<n>We show that the addition of weight decay yields strictly tighter theoretical bounds.<n>We derive the critical batch size for Muon that minimizes the computational cost of training.
  arXiv  Detail & Related papers  (2025-07-02T11:03:13Z)
• A Unified Theory of Stochastic Proximal Point Methods without Smoothness [52.30944052987393]
  Proximal point methods have attracted considerable interest owing to their numerical stability and robustness against imperfect tuning. This paper presents a comprehensive analysis of a broad range of variations of the proximal point method (SPPM)
  arXiv  Detail & Related papers  (2024-05-24T21:09:19Z)
• Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergence [25.717501580080846]
  This paper revisits the convergence of gradient Forward Mirror (SMD) in the contemporary non optimization setting. For the training, we develop provably convergent algorithms for the problem of linear networks.
  arXiv  Detail & Related papers  (2024-02-27T17:56:49Z)
• Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration [122.51142131506639]
  We introduce a precise, fast, and differentiable upper bound for the spectral norm of convolutional layers using circulant matrix theory. We show through a comprehensive set of experiments that our approach outperforms other state-of-the-art methods in terms of precision, computational cost, and scalability. It proves highly effective for the Lipschitz regularization of convolutional neural networks, with competitive results against concurrent approaches.
  arXiv  Detail & Related papers  (2023-05-25T15:32:21Z)
• Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping [69.9674326582747]
  We propose a new accelerated first-order method called clipped-SSTM for smooth convex optimization with heavy-tailed distributed noise in gradients. We prove new complexity that outperform state-of-the-art results in this case. We derive the first non-trivial high-probability complexity bounds for SGD with clipping without light-tails assumption on the noise.
  arXiv  Detail & Related papers  (2020-05-21T17:05:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.