Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

• URL: http://arxiv.org/abs/2606.04754v1
• Date: Wed, 03 Jun 2026 11:37:58 GMT
• Title: Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability
• Abstract summary: Despite growing attention to parameter symmetries, the interplay between parameters, data, and representations remains underexplored.<n>We formalize effective symmetry breaking via neuron identifiability across independent training runs.<n>Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models.
• Score: 42.828413986050045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.

Related papers

• On De-Individuated Neurons: Continuous Symmetries Enable Dynamic Topologies [0.0]
  This paper introduces a novel methodology for computation by leveraging a new symmetry-principled class of primitives, isotropic activation functions.<n>It enables real-time neuronal growth and shrinkage of the architectures in response to task demand.
  arXiv  Detail & Related papers  (2026-02-26T12:47:14Z)
• Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning [73.18052192964349]
  We develop a theoretical framework that explains how discrete symbolic structures can emerge naturally from continuous neural network training dynamics.<n>By lifting neural parameters to a measure space and modeling training as Wasserstein gradient flow, we show that under geometric constraints, the parameter measure $mu_t$ undergoes two concurrent phenomena.
  arXiv  Detail & Related papers  (2025-06-26T22:40:30Z)
• Neural Contraction Metrics with Formal Guarantees for Discrete-Time Nonlinear Dynamical Systems [17.905596843865705]
  Contraction metrics provide a powerful framework for analyzing stability, robustness, and convergence of various dynamical systems.<n>However, identifying these metrics for complex nonlinear systems remains an open challenge due to the lack of effective tools.<n>This paper develops verifiable contraction metrics for discrete scalable nonlinear systems.
  arXiv  Detail & Related papers  (2025-04-23T21:27:32Z)
• Learning Broken Symmetries with Approximate Invariance [1.0485739694839669]
  In many cases, the exact underlying symmetry is present only in an idealized dataset, and is broken in actual data.<n>Standard approaches, such as data augmentation or equivariant networks fail to represent the nature of the full, broken symmetry.<n>We propose a learning model which balances the generality and performance of unconstrained networks with the rapid learning of constrained networks.
  arXiv  Detail & Related papers  (2024-12-25T04:29:04Z)
• Connectivity structure and dynamics of nonlinear recurrent neural networks [41.038855001225826]
  We develop a theory to analyze how connectivity structure shape high-dimensional collective activity in neural networks.<n>We show that connectivity structure can be invisible in single-neuron activities while dramatically shaping collective activity.
  arXiv  Detail & Related papers  (2024-09-03T15:08:37Z)
• The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof [50.49582712378289]
  We investigate the impact of neural parameter symmetries by introducing new neural network architectures. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries.
  arXiv  Detail & Related papers  (2024-05-30T16:32:31Z)
• Symmetry Induces Structure and Constraint of Learning [0.0]
  We unveil the importance of the loss function symmetries in affecting, if not deciding, the learning behavior of machine learning models. Common instances of mirror symmetries in deep learning include rescaling, rotation, and permutation symmetry. We show that the theoretical framework can explain intriguing phenomena, such as the loss of plasticity and various collapse phenomena in neural networks.
  arXiv  Detail & Related papers  (2023-09-29T02:21:31Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Modeling Implicit Bias with Fuzzy Cognitive Maps [0.0]
  This paper presents a Fuzzy Cognitive Map model to quantify implicit bias in structured datasets. We introduce a new reasoning mechanism equipped with a normalization-like transfer function that prevents neurons from saturating.
  arXiv  Detail & Related papers  (2021-12-23T17:04:12Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.