Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials

• URL: http://arxiv.org/abs/2607.02499v1
• Date: Thu, 02 Jul 2026 17:57:31 GMT
• Title: Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
• Abstract summary: We implement and compare a class of recently proposed matrix-structureds, including Muon, SOAP, and the hybrid SOAP-Muon, for training NequIP and Allegro MLIP models.<n>We find that theses can substantially outperform Adam in both convergence speed and final accuracy.
• Score: 0.5714512533033075
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning interatomic potentials (MLIPs) have become a hallmark of AI for scientific simulation. While efforts on new architectures and datasets have led to increasingly accurate and general models, the choice of optimizer for training has largely remained unexplored, defaulting to Adam and its variants in the community. Here, we implement and systematically compare a class of recently proposed matrix-structured optimizers, including Muon, SOAP, and the hybrid SOAP-Muon, for training NequIP and Allegro MLIP models. We find that these optimizers can substantially outperform Adam in both convergence speed and final accuracy. SOAP and SOAP-Muon emerge as robust and consistently strong methods, while Muon only provides partial gains relative to Adam. The improvements are particularly pronounced under partial force supervision. Our results indicate that optimizer choice is an overlooked yet impactful design axis for MLIPs.

Related papers

• SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales [48.03325171473355]
  Higher-order LLMs such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges have limited adoption at scale.<n>We adapt and enhance preconditioned methods to overcome the practical challenges of large-scale LLM pretraining regimes.
  arXiv  Detail & Related papers  (2026-07-13T21:47:08Z)
• Benchmarking Optimizers for MLPs in Tabular Deep Learning [26.245361419936135]
  AdamW is the go-to training technique for training tabular deep learning models.<n>We benchmark 15 techniques on 17 datasets for training-based models in the standard supervised learning setting.<n>Our main finding is that the Muon consistently outperforms AdamW, and thus should be considered a strong and practical choice for practitioners and researchers.
  arXiv  Detail & Related papers  (2026-04-16T17:57:02Z)
• NorMuon: Making Muon more efficient and scalable [71.49702449498085]
  We propose NorMuon (Neuron-wise Normalized Muon) as a successor to Adam.<n>We show NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting.
  arXiv  Detail & Related papers  (2025-10-07T01:13:41Z)
• Muon Outperforms Adam in Tail-End Associative Memory Learning [118.98991042050532]
  We show that Muon consistently achieves balanced learning across classes regardless of feature embeddings.<n>Our empirical observations and theoretical analyses reveal Muon's core advantage: its update rule aligns with the outer-product structure of linear associative memories.
  arXiv  Detail & Related papers  (2025-09-30T10:04:08Z)
• Conda: Column-Normalized Adam for Training Large Language Models Faster [70.66067959375748]
  Column-Normalized Adam (Conda) is a novel approach to large language models (LLMs)<n>Conda projects updates into a subspace and applies column-wise second moment normalization based on the projected gradients.<n>Experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training.
  arXiv  Detail & Related papers  (2025-09-29T02:58:19Z)
• AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training [22.58304858379219]
  We introduce AdamS, a simple yet effective alternative to Adam for large language model (LLM) pretraining and post-training.<n>By leveraging a novel denominator, i.e., the root of weighted sum of squares of the momentum and the current gradient, AdamS eliminates the need for second-moment estimates.<n>AdamS is efficient, matching the memory and compute footprint of SGD with momentum while delivering superior optimization performance.
  arXiv  Detail & Related papers  (2025-05-22T08:16:48Z)
• Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs [111.69640966866059]
  Sparse large language models (LLMs) with Mixture of Experts (MoE) and close to a trillion parameters are dominating the realm of most capable language models.<n>In this paper, we aim to uncover a recipe to harness such scale on Ascend NPUs.<n>The key goals are better usage of the computing resources under the dynamic sparse model structures and materializing the expected performance gain on the actual hardware.
  arXiv  Detail & Related papers  (2025-05-07T15:46:36Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• CAdam: Confidence-Based Optimization for Online Learning [41.022196390765714]
  We introduce CAdam, a confidence-based optimization strategy that assesses the consistency between the momentum and the gradient for each parameter dimension before deciding on updates.<n>CAdam significantly enhances model performance compared to Adam, leading to substantial increases in the system's gross merchandise volume (GMV)<n>In large-scale A/B testing, CAdam significantly enhances model performance compared to Adam, leading to substantial increases in the system's gross merchandise volume (GMV)
  arXiv  Detail & Related papers  (2024-11-29T12:00:27Z)
• MARS: Unleashing the Power of Variance Reduction for Training Large Models [56.67982828148859]
  We propose a unified training framework for deep neural networks.<n>We introduce three instances of MARS that leverage preconditioned gradient optimization.<n>Results indicate that the implementation of MARS consistently outperforms Adam.
  arXiv  Detail & Related papers  (2024-11-15T18:57:39Z)
• AdamL: A fast adaptive gradient method incorporating loss function [1.6025685183216696]
  We propose AdamL, a novel variant of the Adam, that takes into account the loss function information to attain better results. We show that AdamL achieves either the fastest convergence or the lowest objective function values when compared to Adam, EAdam, and AdaBelief. In the case of vanilla convolutional neural networks, AdamL stands out from the other Adam's variants and does not require the manual adjustment of the learning rate during the later stage of the training.
  arXiv  Detail & Related papers  (2023-12-23T16:32:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.