When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study

• URL: http://arxiv.org/abs/2605.26929v2
• Date: Fri, 29 May 2026 15:42:11 GMT
• Title: When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
• Abstract summary: Adrial training (AT) remains one of the most reliable empirical defenses against adversarial attacks.<n>SGD remains the default optimization choice for AT, whereas adaptives often improve standard training but may yield inferior robustness.<n>Muon has achieved notable success in large-scale training at a memory cost comparable to SGD.
• Score: 12.606171332119908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial training (AT) remains one of the most reliable empirical defenses against adversarial attacks. Its robustness critically depends on how the underlying min-max objective is optimized. In practice, Stochastic Gradient Descent (SGD) optimizer remains the default optimization choice for AT, whereas adaptive optimizers often improve standard training but may yield inferior robustness. Recently, the Muon optimizer, which orthogonalizes matrix-valued updates via an approximate polar decomposition, has achieved notable success in large-scale training at a memory cost comparable to SGD. This raises a security-relevant question: \textit{can orthogonalized optimization improve AT under strong and heterogeneous threat models?} Focusing on this problem, we conduct a comprehensive theoretical and empirical study. Theoretically, we show that Muon imposes a spectral-norm stability ceiling on matrix updates, limiting uncontrolled spectral growth in the training dynamics without explicitly shrinking the learned weights. Empirically, across five architectures and three $\ell_p$ threat models ($\ell_\infty$, $\ell_1$, $\ell_2$) and their union, Muon is competitive with SGD on CNNs and substantially outperforms AdamW on both CNNs and ViTs. These results identify optimizer geometry as a security-relevant factor in adversarial training, while clarifying the empirical regimes in which orthogonalized updates are beneficial. Overall, our findings highlight optimizer design as a security-critical component of AT.

Related papers

• A Defense of the Quadratic Model [71.57026347416729]
  We stress test the simplest possible model of optimization -- the quadratic model.<n>We show that it can be predictive surprisingly in an LLM setting with 150M parameters and 3B training tokens.
  arXiv  Detail & Related papers  (2026-07-23T18:02:03Z)
• ROOT: Robust Orthogonalized Optimizer for Neural Network Training [47.05662448082334]
  Large language models (LLMs) remain a critical challenge, particularly as model scaling exacerbates sensitivity to imprecision and training instability.<n>We develop a dimension-robustization scheme that enhances robustness through iterations tailored to specific matrix sizes.<n>Second, we introduce an optimization-robustization framework that suppresses outliers noise while preserving meaningful directions.
  arXiv  Detail & Related papers  (2025-11-25T18:48:05Z)
• Beyond the Ideal: Analyzing the Inexact Muon Update [54.70108543057578]
  We show first analysis of the inexactized update at Muon's core.<n>We reveal a fundamental coupling between this inexactness and the optimal step size and momentum.
  arXiv  Detail & Related papers  (2025-10-22T18:01:07Z)
• Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning [25.53799024782883]
  Large language model (LLM) unlearning aims to surgically remove the influence of undesired data or knowledge from an existing model.<n>Recent findings reveal that unlearning manipulations such as weight quantization or fine-tuning can quickly neutralize the intended forgetting.
  arXiv  Detail & Related papers  (2025-10-01T10:50:14Z)
• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories [58.988535279557546]
  We introduce textbf sycophancy Mitigation through Adaptive Reasoning Trajectories.<n>We show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-09-20T17:09:14Z)
• Optimizers Qualitatively Alter Solutions And We Should Leverage This [62.662640460717476]
  Deep Neural Networks (DNNs) can not guarantee convergence to a unique global minimum of the loss when using only local information, such as SGD.<n>We argue that the community should aim at understanding the biases of already existing methods, as well as aim to build new DNNs with the explicit intent of inducing certain properties of the solution.
  arXiv  Detail & Related papers  (2025-07-16T13:33:31Z)
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning [45.64632177923583]
  This work studies the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences.<n>Despite the increasing demand for unlearning, a technically-grounded optimization framework is lacking.<n>We propose a simple yet effective unlearning optimization framework, called SimNPO, showing that simplicity' in removing the reliance on a reference model benefits unlearning.
  arXiv  Detail & Related papers  (2024-10-09T17:58:12Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)
• Bridging the Gap Between Adversarial Robustness and Optimization Bias [28.56135898767349]
  Adrial robustness is an open challenge in deep learning, most often tackled using adversarial training. We show that it is possible to achieve both perfect standard accuracy and a certain degree of robustness without a trade-off. In particular, we characterize the robustness of linear convolutional models, showing that they resist attacks subject to a constraint on the Fourier-$ell_infty$ norm.
  arXiv  Detail & Related papers  (2021-02-17T16:58:04Z)
• Improved Adversarial Training via Learned Optimizer [101.38877975769198]
  We propose a framework to improve the robustness of adversarial training models. By co-training's parameters model's weights, the proposed framework consistently improves robustness and steps adaptively for update directions.
  arXiv  Detail & Related papers  (2020-04-25T20:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.