Schattor: Schatten-family methods for deep learning optimization

• URL: http://arxiv.org/abs/2606.15702v1
• Date: Sun, 14 Jun 2026 09:31:59 GMT
• Title: Schattor: Schatten-family methods for deep learning optimization
• Abstract summary: We propose a family of adaptive firstorder methods based on Schatten norms.<n>We establish dimension-free station guarantees for methods Schattor and SGD.<n>We also develop multiblock extensions that adaptively balance block-wise optimization progress.
• Score: 13.92796349861358
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis. Motivated by the limitations of SGD and the success of adaptive optimizers, we propose {\it Schattor}, a family of adaptive first-order methods based on Schatten norms. Schattor unifies SGD and the recently proposed matrix-variate adaptive optimizer Muon within a single Schatten-norm-based framework. We establish dimension-free stationarity guarantees for methods in the Schattor family for stochastic matrix optimization problems via a novel matrix martingale moment bound. We also develop multi-block extensions that adaptively balance block-wise optimization progress and prove dimension-free stationarity guarantees in this more general setting.

Related papers

• Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning [46.539149169728624]
  A major obstacle in large-scale fine-tuning is the memory overhead of backpropagation.<n>$texttAdaNAGED$ is a method that unifies gradient-free training, adaptive tuning, and non-Euclidean update geometry.
  arXiv  Detail & Related papers  (2026-06-12T21:46:54Z)
• Diff3R: Feed-forward 3D Gaussian Splatting with Uncertainty-aware Differentiable Optimization [76.38917994186733]
  Diff3R is a novel framework that bridges feed-forward prediction and test-time optimization.<n>We show that it can be seamlessly integrated into existing feed-forward 3DGS architectures for both pose-given and pose-free methods.
  arXiv  Detail & Related papers  (2026-04-01T15:40:20Z)
• Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization [54.723834588133165]
  We study online linear optimization with matrix variables by the operatorAML, a setting where the geometry renders designing datadependent and efficient adaptive algorithms challenging.<n>We instantiate this framework with two efficient methods that avoid projections.<n>We show both methods admit closed-form updates match one-sided Shampoo's regret up to a constant factor, while significantly reducing computational cost.
  arXiv  Detail & Related papers  (2026-02-09T03:09:47Z)
• Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods [50.070182958880146]
  We propose a unified framework generalizing descent, quasi-Newton methods, and adaptive methods through the novel notion of preconditioned matrix norms.<n>Within this framework, we provide the first systematic treatment of affine and scale invariance in the matrix- parameterized setting.<n>We introduce two new methods, $ttMuAdam$ and $texttMuAdam-SANIA$, which combine the spectral geometry of Muon with Adam-style preconditioning.
  arXiv  Detail & Related papers  (2025-10-12T19:39:41Z)
• Learning to optimize with guarantees: a complete characterization of linearly convergent algorithms [1.4747234049753448]
  In high-stakes engineering applications, optimization algorithms must come with provable provablecase guarantees over a mathematically defined class of problems.<n>We describe the class of algorithms that achieve linear convergence for classes of nonsmooth composite optimization problems.
  arXiv  Detail & Related papers  (2025-08-01T16:56:42Z)
• Differential Evolution for Grassmann Manifold Optimization: A Projection Approach [0.0]
  We propose a novel evolutionary algorithm for real-valued objective functions defined on the Grassmann manifold Gr(k,n)<n>Our approach incorporates a projection that maps vectors onto the manifold via decomposition.
  arXiv  Detail & Related papers  (2025-03-27T21:04:19Z)
• Training Deep Learning Models with Norm-Constrained LMOs [56.00317694850397]
  We propose a new family of algorithms that uses the linear minimization oracle (LMO) to adapt to the geometry of the problem.<n>We demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam.
  arXiv  Detail & Related papers  (2025-02-11T13:10:34Z)
• Improved Approximation Algorithms for Low-Rank Problems Using Semidefinite Optimization [2.0052993723676895]
  We construct an analogous relax-then-round strategy for low-rank optimization problems.<n>For a given $n times m$ semi-orthogonal matrix, we derive a purely multiplicative approximation ratio for our algorithm.<n>We extend our approach to generic low-rank optimization problems by developing new semidefinite relaxations.
  arXiv  Detail & Related papers  (2025-01-06T11:31:41Z)
• Adaptive Stochastic Optimisation of Nonconvex Composite Objectives [2.1700203922407493]
  We propose and analyse a family of generalised composite mirror descent algorithms. With adaptive step sizes, the proposed algorithms converge without requiring prior knowledge of the problem. We exploit the low-dimensional structure of the decision sets for high-dimensional problems.
  arXiv  Detail & Related papers  (2022-11-21T18:31:43Z)
• Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs [113.8752163061151]
  We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs)<n>We propose the underlineperiodically underlinerestarted underlineoptimistic underlinepolicy underlineoptimization algorithm (PROPO)<n>PROPO features two mechanisms: sliding-window-based policy evaluation and periodic-restart-based policy improvement.
  arXiv  Detail & Related papers  (2021-10-18T02:33:20Z)
• Private Adaptive Gradient Methods for Convex Optimization [32.3523019355048]
  We propose and analyze differentially private variants of a Gradient Descent (SGD) algorithm with adaptive stepsizes. We provide upper bounds on the regret of both algorithms and show that the bounds are (worst-case) optimal.
  arXiv  Detail & Related papers  (2021-06-25T16:46:45Z)
• EOS: a Parallel, Self-Adaptive, Multi-Population Evolutionary Algorithm for Constrained Global Optimization [68.8204255655161]
  EOS is a global optimization algorithm for constrained and unconstrained problems of real-valued variables. It implements a number of improvements to the well-known Differential Evolution (DE) algorithm. Results prove that EOSis capable of achieving increased performance compared to state-of-the-art single-population self-adaptive DE algorithms.
  arXiv  Detail & Related papers  (2020-07-09T10:19:22Z)
• Convergence of adaptive algorithms for weakly convex constrained optimization [59.36386973876765]
  We prove the $mathcaltilde O(t-1/4)$ rate of convergence for the norm of the gradient of Moreau envelope. Our analysis works with mini-batch size of $1$, constant first and second order moment parameters, and possibly smooth optimization domains.
  arXiv  Detail & Related papers  (2020-06-11T17:43:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.