Noether: The More Things Change, the More Stay the Same

• URL: http://arxiv.org/abs/2104.05508v1
• Date: Mon, 12 Apr 2021 14:41:05 GMT
• Title: Noether: The More Things Change, the More Stay the Same
• Authors: Grzegorz G{\l}uch, R\"udiger Urbanke
• Abstract summary: Noether's celebrated theorem states that symmetry leads to conserved quantities. In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. Symmetry can be thought of as one further important tool in understanding the performance of neural networks under gradient descent.
• Score: 1.14219428942199
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Symmetries have proven to be important ingredients in the analysis of neural networks. So far their use has mostly been implicit or seemingly coincidental. We undertake a systematic study of the role that symmetry plays. In particular, we clarify how symmetry interacts with the learning algorithm. The key ingredient in our study is played by Noether's celebrated theorem which, informally speaking, states that symmetry leads to conserved quantities (e.g., conservation of energy or conservation of momentum). In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. E.g., we show that symmetry of activation functions leads to boundedness of weight matrices, for the specific case of linear activations it leads to balance equations of consecutive layers, data augmentation leads to gradient paths that have "momentum"-type restrictions, and time symmetry leads to a version of the Neural Tangent Kernel. Symmetry alone does not specify the optimization path, but the more symmetries are contained in the model the more restrictions are imposed on the path. Since symmetry also implies over-parametrization, this in effect implies that some part of this over-parametrization is cancelled out by the existence of the conserved quantities. Symmetry can therefore be thought of as one further important tool in understanding the performance of neural networks under gradient descent.

Related papers

• Lie Point Symmetry and Physics Informed Networks [59.56218517113066]
  We propose a loss function that informs the network about Lie point symmetries in the same way that PINN models try to enforce the underlying PDE through a loss function. Our symmetry loss ensures that the infinitesimal generators of the Lie group conserve the PDE solutions. Empirical evaluations indicate that the inductive bias introduced by the Lie point symmetries of the PDEs greatly boosts the sample efficiency of PINNs.
  arXiv  Detail & Related papers  (2023-11-07T19:07:16Z)
• Learning Layer-wise Equivariances Automatically using Gradients [66.81218780702125]
  Convolutions encode equivariance symmetries into neural networks leading to better generalisation performance. symmetries provide fixed hard constraints on the functions a network can represent, need to be specified in advance, and can not be adapted. Our goal is to allow flexible symmetry constraints that can automatically be learned from data using gradients.
  arXiv  Detail & Related papers  (2023-10-09T20:22:43Z)
• Symmetry Induces Structure and Constraint of Learning [0.0]
  We unveil the importance of the loss function symmetries in affecting, if not deciding, the learning behavior of machine learning models. Common instances of mirror symmetries in deep learning include rescaling, rotation, and permutation symmetry. We show that the theoretical framework can explain intriguing phenomena, such as the loss of plasticity and various collapse phenomena in neural networks.
  arXiv  Detail & Related papers  (2023-09-29T02:21:31Z)
• Identifying the Group-Theoretic Structure of Machine-Learned Symmetries [41.56233403862961]
  We propose methods for examining and identifying the group-theoretic structure of such machine-learned symmetries. As an application to particle physics, we demonstrate the identification of the residual symmetries after the spontaneous breaking of non-Abelian gauge symmetries.
  arXiv  Detail & Related papers  (2023-09-14T17:03:50Z)
• On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis [38.92081817503126]
  We study discrete morphological symmetries of dynamical systems. These symmetries arise from the presence of one or more planes/axis of symmetry in the system's morphology. We exploit these symmetries using data augmentation and $G$-equivariant neural networks.
  arXiv  Detail & Related papers  (2023-02-21T04:10:16Z)
• Symmetry protected entanglement in random mixed states [0.0]
  We study the effect of symmetry on tripartite entanglement properties of typical states in symmetric sectors of Hilbert space. In particular, we consider Abelian symmetries and derive an explicit expression for the logarithmic entanglement negativity of systems with $mathbbZ_N$ and $U(1)$ symmetry groups.
  arXiv  Detail & Related papers  (2021-11-30T19:00:07Z)
• Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning [7.310043452300738]
  In nature, symmetry governs regularities, while symmetry breaking brings texture. Recent experiments suggest that the symmetry of the loss function is closely related to the learning performance. We pose symmetry breaking as a new design principle by considering the symmetry of the learning rule in addition to the loss function.
  arXiv  Detail & Related papers  (2021-05-06T14:36:10Z)
• Symmetry Breaking in Symmetric Tensor Decomposition [44.181747424363245]
  We consider the nonsymmetry problem associated with computing the points rank decomposition of symmetric tensors. We show that critical points the loss function is detected by standard methods.
  arXiv  Detail & Related papers  (2021-03-10T18:11:22Z)
• Beyond Procrustes: Balancing-Free Gradient Descent for Asymmetric Low-Rank Matrix Sensing [36.96922859748537]
  Low-rank matrix estimation plays a central role in various applications across science and engineering. Existing approaches rely on adding a metric regularization term to balance the scale of the two matrix factors. In this paper, we provide a theoretical justification for the performance in recovering a low-rank matrix from a small number of linear measurements.
  arXiv  Detail & Related papers  (2021-01-13T15:03:52Z)
• Latent symmetry induced degeneracies [0.0]
  We develop an approach to explain degeneracies by tracing them back to symmetries of an isospectral effective Hamiltonian. As an application, we relate the degeneracies induced by the rotation symmetry of a real Hamiltonian to a non-abelian latent symmetry group.
  arXiv  Detail & Related papers  (2020-11-26T17:37:30Z)
• Finding Symmetry Breaking Order Parameters with Euclidean Neural Networks [2.735801286587347]
  We demonstrate that symmetry equivariant neural networks uphold Curie's principle and can be used to articulate many symmetry-relevant scientific questions into simple optimization problems. We prove these properties mathematically and demonstrate them numerically by training a Euclidean symmetry equivariant neural network to learn symmetry-breaking input to deform a square into a rectangle and to generate octahedra tilting patterns in perovskites.
  arXiv  Detail & Related papers  (2020-07-04T17:24:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.