How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks

• URL: http://arxiv.org/abs/2605.27662v1
• Date: Tue, 26 May 2026 20:25:44 GMT
• Title: How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks
• Abstract summary: Equivariant neural networks encode geometric symmetries by construction, yet they are often difficult to optimize and can underperform less constrained architectures.<n>We study this direction by comparing Muon and Adam across several equivariant and geometric architectures under pointcloud and molecular learning settings.<n>We then analyze the trained ModelNet40 checkpoints through Hessian estimates, loss surface visualizations, and spectral properties of learned weights and intermediate representations.
• Score: 3.449933410012168
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Equivariant neural networks encode geometric symmetries by construction, yet they are often difficult to optimize and can underperform less constrained architectures. A growing body of work addresses this through architectural modifications such as constraint relaxation or approximate equivariance, while the role of the optimizer remains comparatively underexplored. We study this direction by comparing Muon and Adam across several equivariant and geometric architectures under pointcloud and molecular learning settings. On ModelNet40, where the comparison is clearest, Muon consistently improves over Adam across all architectures considered. We then analyze the trained ModelNet40 checkpoints through Hessian estimates, loss surface visualizations, and spectral properties of learned weights and intermediate representations. The checkpoints reached by Muon have larger Hessian curvature summaries but more regular loss surfaces, and their learned weights and representations have higher stable and effective ranks. These observations suggest that the interaction between optimizer design and geometric inductive bias deserves further attention from the community.

Related papers

• On the Geometric Structure of Layer Updates in Deep Language Models [0.0]
  We study the geometric structure of layer updates in deep language models.<n>We show that layerwise updates admit a decomposition into a dominant tokenwise component and a residual that is not captured by restricted tokenwise function classes.
  arXiv  Detail & Related papers  (2026-04-02T18:44:34Z)
• The Effect of Architecture During Continual Learning [2.1485350418225244]
  We introduce a mathematical framework that jointly models architecture and weights in a Sobolev space.<n>We prove that learning only model weights is insufficient to mitigate catastrophic forgetting under distribution shifts.<n> Empirical studies across regression and classification problems, including feedforward, convolutional, and graph neural networks, demonstrate that learning the optimal architecture and weights simultaneously yields substantially improved performance.
  arXiv  Detail & Related papers  (2026-01-27T16:29:42Z)
• Soft Geometric Inductive Bias for Object Centric Dynamics [36.337338384532636]
  We propose object-centric world models built with geometric algebra neural networks.<n>We show that the soft inductive bias of our models results in better performance in terms of physical fidelity.
  arXiv  Detail & Related papers  (2025-12-17T14:40:37Z)
• Symmetry-Aware Fully-Amortized Optimization with Scale Equivariant Graph Metanetworks [39.69046654861533]
  We show that Scale Equivariant Graph Metanetworks (ScaleGMNs) enable single-shot fine-tuning of existing models.<n>We show that the gauge freedom induced by scaling symmetries is strictly smaller in convolutional neural networks than in multi-layer perceptrons.<n>Our findings underscore the potential of symmetry-aware metanetworks as a powerful approach for efficient and generalizable neural network optimization.
  arXiv  Detail & Related papers  (2025-10-09T14:51:15Z)
• Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion [55.95767828747407]
  In domains such as molecular and protein generation, physical systems exhibit inherent symmetries that are critical to model.<n>We present a framework that reduces training variance and provides a provably lower-variance gradient estimator.<n>We also present a practical implementation of this estimator incorporating the loss and sampling procedure through a method we call Orbit Diffusion.
  arXiv  Detail & Related papers  (2025-02-14T03:26:57Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Exploring the design space of deep-learning-based weather forecasting systems [56.129148006412855]
  This paper systematically analyzes the impact of different design choices on deep-learning-based weather forecasting systems. We study fixed-grid architectures such as UNet, fully convolutional architectures, and transformer-based models. We propose a hybrid system that combines the strong performance of fixed-grid models with the flexibility of grid-invariant architectures.
  arXiv  Detail & Related papers  (2024-10-09T22:25:50Z)
• Enhancing lattice kinetic schemes for fluid dynamics with Lattice-Equivariant Neural Networks [79.16635054977068]
  We present a new class of equivariant neural networks, dubbed Lattice-Equivariant Neural Networks (LENNs) Our approach develops within a recently introduced framework aimed at learning neural network-based surrogate models Lattice Boltzmann collision operators. Our work opens towards practical utilization of machine learning-augmented Lattice Boltzmann CFD in real-world simulations.
  arXiv  Detail & Related papers  (2024-05-22T17:23:15Z)
• The Lie Derivative for Measuring Learned Equivariance [84.29366874540217]
  We study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. We find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities. For example, transformers can be more equivariant than convolutional neural networks after training.
  arXiv  Detail & Related papers  (2022-10-06T15:20:55Z)
• SmoothNets: Optimizing CNN architecture design for differentially private deep learning [69.10072367807095]
  DPSGD requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. We distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training.
  arXiv  Detail & Related papers  (2022-05-09T07:51:54Z)
• Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification [8.976788958300766]
  This work investigates the problem of disentangling the role of the neural structure and its edge weights. We show that well-trained architectures may not need any link-specific fine-tuning of the weights. We use a novel and computationally efficient method that translates the hard architecture-search problem into a feasible optimization problem.
  arXiv  Detail & Related papers  (2020-09-11T11:22:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.