Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

• URL: http://arxiv.org/abs/2607.07845v1
• Date: Wed, 08 Jul 2026 18:27:16 GMT
• Title: Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks
• Abstract summary: Hessian of the training loss governs the local geometry of the loss landscape.<n>We argue that the bulk consists of the weakly lifted pseudo-Goldstone modes of the continuous symmetries of the network parametrization.
• Score: 3.437656066916039
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive. We argue that the bulk consists of the weakly lifted pseudo-Goldstone modes of the continuous symmetries of the network parametrization. In deep linear networks these symmetries are exact: they generate flat directions and hence exact zero modes, whose eigenvectors we construct explicitly. Introducing a ReLU nonlinearity as a perturbation, we show that it breaks these symmetries weakly and explicitly. Resolving the spectrum at the level of eigenvectors, we find that the high-curvature directions are orthogonal to the symmetry subspace, while the bulk lies almost entirely within it. We demonstrate the mechanism in a two-layer ReLU student--teacher model and in a network trained on CIFAR-10. A convolutional example demonstrates that the same diagnostic extends beyond fully connected layers. Together, these results link the Hessian bulk to weakly broken symmetries and clarify the origin of near-zero modes.

Related papers

• The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization [57.37943479039033]
  We study how architectural inductive bias reshapes the implicit regularization induced by the edge-of-stability phenomenon in gradient descent.<n>We show that locality and weight sharing fundamentally change this picture.
  arXiv  Detail & Related papers  (2026-03-05T04:50:51Z)
• Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU [3.2063364612188416]
  We study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation.<n>We detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter $alpha$ varies.
  arXiv  Detail & Related papers  (2025-10-29T01:00:07Z)
• Trading Mathematical for Physical Simplicity: Bialgebraic Structures in Matrix Product Operator Symmetries [20.76275069383104]
  We show that simple quantum spin chains of physical interest are not included in the rigid framework of fusion categories and weak Hopf algebras.<n>We show that our work provides a bridge between well-understood topological defect symmetries and those that arise in more realistic models.
  arXiv  Detail & Related papers  (2025-09-03T18:01:22Z)
• Mixed-state phases from local reversibility [3.3599784433577886]
  We propose a refined definition of mixed-state phase equivalence based on locally reversible channel circuits.<n>We show that such circuits preserve topological degeneracy and the locality of all operators including both strong and weak symmetries.
  arXiv  Detail & Related papers  (2025-07-03T03:59:07Z)
• Generalized Linear Mode Connectivity for Transformers [87.32299363530996]
  A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths.<n>Prior work has predominantly focused on neuron re-ordering through permutations, but such approaches are limited in scope.<n>We introduce a unified framework that captures four symmetry classes: permutations, semi-permutations, transformations, and general invertible maps.<n>This generalization enables, for the first time, the discovery of low- and zero-barrier linear paths between independently trained Vision Transformers and GPT-2 models.
  arXiv  Detail & Related papers  (2025-06-28T01:46:36Z)
• A Signed Graph Approach to Understanding and Mitigating Oversmoothing in GNNs [54.62268052283014]
  We present a unified theoretical perspective based on the framework of signed graphs.<n>We show that many existing strategies implicitly introduce negative edges that alter message-passing to resist oversmoothing.<n>We propose Structural Balanced Propagation (SBP), a plug-and-play method that assigns signed edges based on either labels or feature similarity.
  arXiv  Detail & Related papers  (2025-02-17T03:25:36Z)
• Black Boxes and Looking Glasses: Multilevel Symmetries, Reflection Planes, and Convex Optimization in Deep Networks [46.337104465755075]
  We show that training deep neural networks (DNNs) with absolute value activation and arbitrary input dimension can be formulated as equivalent convex Lasso problems. This formulation reveals geometric structures encoding symmetry in neural networks.
  arXiv  Detail & Related papers  (2024-10-05T20:09:07Z)
• Localization with non-Hermitian off-diagonal disorder [0.0]
  We discuss a non-Hermitian system governed by random nearest-neighbour tunnellings.<n>A physical situation of completely real eigenspectrum arises owing to the Hamiltonian's tridiagonal matrix structure.<n>The off-diagonal disorder leads the non-Hermitian system to a delocalization-localization crossover in finite systems.
  arXiv  Detail & Related papers  (2023-10-20T18:02:01Z)
• Symmetry Induces Structure and Constraint of Learning [0.0]
  We unveil the importance of the loss function symmetries in affecting, if not deciding, the learning behavior of machine learning models. Common instances of mirror symmetries in deep learning include rescaling, rotation, and permutation symmetry. We show that the theoretical framework can explain intriguing phenomena, such as the loss of plasticity and various collapse phenomena in neural networks.
  arXiv  Detail & Related papers  (2023-09-29T02:21:31Z)
• Symmetries, flat minima, and the conserved quantities of gradient flow [20.12938444246729]
  We present a framework for finding continuous symmetries in the parameter space, which carves out low-loss valleys. To generalize this framework to nonlinear neural networks, we introduce a novel set of nonlinear, data-dependent symmetries.
  arXiv  Detail & Related papers  (2022-10-31T10:55:30Z)
• Training invariances and the low-rank phenomenon: beyond linear networks [44.02161831977037]
  We show that when one trains a deep linear network with logistic or exponential loss on linearly separable data, the weights converge to rank-$1$ matrices. This is the first time a low-rank phenomenon is proven rigorously for nonlinear ReLU-activated feedforward networks. Our proof relies on a specific decomposition of the network into a multilinear function and another ReLU network whose weights are constant under a certain parameter directional convergence.
  arXiv  Detail & Related papers  (2022-01-28T07:31:19Z)
• Boundary theories of critical matchgate tensor networks [59.433172590351234]
  Key aspects of the AdS/CFT correspondence can be captured in terms of tensor network models on hyperbolic lattices. For tensors fulfilling the matchgate constraint, these have previously been shown to produce disordered boundary states. We show that these Hamiltonians exhibit multi-scale quasiperiodic symmetries captured by an analytical toy model.
  arXiv  Detail & Related papers  (2021-10-06T18:00:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.