Tensorion: A Tensor-Aware Generalization of the Muon Optimizer

• URL: http://arxiv.org/abs/2606.25975v1
• Date: Wed, 24 Jun 2026 15:46:04 GMT
• Title: Tensorion: A Tensor-Aware Generalization of the Muon Optimizer
• Abstract summary: Recent work has shown that exploiting matrix structure can improve dynamics.<n>A notable example is Muon, which performs steepest descent under the spectral norm constraint.<n>We take the next step and introduceion, a tensor-aware that extends Muon's constrained optimization perspective.
• Score: 38.75730408705132
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models. Recent work has shown that exploiting matrix structure can improve optimization dynamics. A notable example is Muon, which performs steepest descent under the spectral norm constraint. We take the next step and introduce Tensorion, a tensor-aware optimizer that extends Muon's constrained optimization perspective from matrices to higher-order tensors. Tensorion is built around a linear minimization oracle (LMO) over a tensor norm ball. The norm is carefully chosen to balance two objectives: tightly bounding the tensor spectral norm, while still keeping the LMO tractable. This LMO becomes computable because it reduces to operations on adaptively selected unfolding matrices. Notably, when restricted to order-2 tensors (i.e., matrices), Tensorion recovers Muon exactly. Experiments on tensor-based computer vision problems suggest that Tensorion can offer improved convergence behavior and more stable gradient updates compared with Adam-based and existing tensor-aware baselines in the evaluated settings.

Related papers

• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors [110.10775872372047]
  We propose emphMagnitude-Direction-Direction (MD) Decoupling.<n>It factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates.<n>Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts models.
  arXiv  Detail & Related papers  (2026-06-24T15:40:26Z)
• Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise [12.977441534320041]
  We show that a scale-invariant first-order method with spectral norm requires $(minm, n-frac3p-2p-1) calls.<n>We prove that a batched method achieves the matching bound of $(minm, n-frac5p2p-2p-2)$ when the norm is spectral and the Hessian is Lipschitz.
  arXiv  Detail & Related papers  (2026-05-18T15:13:18Z)
• Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization [14.136955342888987]
  We reform the AdaGrad update and decompose it into a variance adaptation term and a scale-invariant term.<n>This produces $textbfDeVA$ ($textbfV$ariance $textbfA$daptation), a framework that bridges between vector-based variance adaptation and matrix spectral optimization.
  arXiv  Detail & Related papers  (2026-02-06T17:06:42Z)
• Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers [11.445970271488095]
  We introduce learnable multipliers to learn the optimal scale for applying weight decay to matrix layers.<n>Our method can be viewed as a learnable, more expressive generalization of muP multipliers.<n>It outperforms a well-tuned muP baseline, reduces the computational overhead of tuning, and surfaces practical questions such as forward-pass symmetries and the width-scaling of the learned multipliers.
  arXiv  Detail & Related papers  (2026-01-08T12:41:49Z)
• Low-Rank Tensor Recovery via Variational Schatten-p Quasi-Norm and Jacobian Regularization [49.85875869048434]
  We propose a CP-based low-rank tensor function parameterized by neural networks for implicit neural representation.<n>To achieve sparser CP decomposition, we introduce a variational Schatten-p quasi-norm to prune redundant rank-1 components.<n>For smoothness, we propose a regularization term based on the spectral norm of the Jacobian and Hutchinson's trace estimator.
  arXiv  Detail & Related papers  (2025-06-27T11:23:10Z)
• Decomposable Sparse Tensor on Tensor Regression [1.370633147306388]
  We consider the sparse low rank tensor on tensor regression where predictors $mathcalX$ and responses $mathcalY$ are both high-dimensional tensors. We propose a fast solution based on stagewise search composed by contraction part and generation part which are optimized alternatively.
  arXiv  Detail & Related papers  (2022-12-09T18:16:41Z)
• Softmax-free Linear Transformers [90.83157268265654]
  Vision transformers (ViTs) have pushed the state-of-the-art for visual perception tasks. Existing methods are either theoretically flawed or empirically ineffective for visual recognition. We propose a family of Softmax-Free Transformers (SOFT)
  arXiv  Detail & Related papers  (2022-07-05T03:08:27Z)
• Matrix Completion via Non-Convex Relaxation and Adaptive Correlation Learning [90.8576971748142]
  We develop a novel surrogate that can be optimized by closed-form solutions. We exploit upperwise correlation for completion, and thus an adaptive correlation learning model.
  arXiv  Detail & Related papers  (2022-03-04T08:50:50Z)
• Tensor Completion Made Practical [19.229475414802213]
  We introduce a new variant of alternating minimization, which in turn is inspired by understanding how the progress measures that guide convergence need to be adapted to the tensor setting. We show that our algorithm converges linearly to the true tensors even when the factors are highly correlated and can be implemented in nearly linear time. In contrast, and somewhat surprisingly, we show that the standard version of alternating minimization, without our new twist, can converge at a drastically slower rate in practice.
  arXiv  Detail & Related papers  (2020-06-04T21:09:44Z)
• Spectral Learning on Matrices and Tensors [74.88243719463053]
  We show that tensor decomposition can pick up latent effects that are missed by matrix methods. We also outline computational techniques to design efficient tensor decomposition methods.
  arXiv  Detail & Related papers  (2020-04-16T22:53:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.