Noether: The More Things Change, the More Stay the Same

• URL: http://arxiv.org/abs/2104.05508v1
• Date: Mon, 12 Apr 2021 14:41:05 GMT
• Title: Noether: The More Things Change, the More Stay the Same
• Authors: Grzegorz G{\l}uch, R\"udiger Urbanke
• Abstract summary: Noether's celebrated theorem states that symmetry leads to conserved quantities. In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. Symmetry can be thought of as one further important tool in understanding the performance of neural networks under gradient descent.
• Score: 1.14219428942199
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Symmetries have proven to be important ingredients in the analysis of neural networks. So far their use has mostly been implicit or seemingly coincidental. We undertake a systematic study of the role that symmetry plays. In particular, we clarify how symmetry interacts with the learning algorithm. The key ingredient in our study is played by Noether's celebrated theorem which, informally speaking, states that symmetry leads to conserved quantities (e.g., conservation of energy or conservation of momentum). In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. E.g., we show that symmetry of activation functions leads to boundedness of weight matrices, for the specific case of linear activations it leads to balance equations of consecutive layers, data augmentation leads to gradient paths that have "momentum"-type restrictions, and time symmetry leads to a version of the Neural Tangent Kernel. Symmetry alone does not specify the optimization path, but the more symmetries are contained in the model the more restrictions are imposed on the path. Since symmetry also implies over-parametrization, this in effect implies that some part of this over-parametrization is cancelled out by the existence of the conserved quantities. Symmetry can therefore be thought of as one further important tool in understanding the performance of neural networks under gradient descent.

Related papers

• The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof [50.49582712378289]
  We investigate the impact of neural parameter symmetries by introducing new neural network architectures. We develop two methods, with some provable guarantees, of modifying standard neural networks to reduce parameter space symmetries. Our experiments reveal several interesting observations on the empirical impact of parameter symmetries.
  arXiv  Detail & Related papers  (2024-05-30T16:32:31Z)
• Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent [8.347295051171525]
  We show that gradient noise creates a systematic interplay of parameters $theta$ along the degenerate direction to a unique-independent fixed point $theta*$. These points are referred to as the it noise equilibria because, at these points, noise contributions from different directions are balanced and aligned. We show that the balance and alignment of gradient noise can serve as a novel alternative mechanism for explaining important phenomena such as progressive sharpening/flattening and representation formation within neural networks.
  arXiv  Detail & Related papers  (2024-02-11T13:00:04Z)
• Asymmetry activation and its relation to coherence under permutation operation [53.64687146666141]
  A Dicke state and its decohered state are invariant for permutation. When another qubits state to each of them is attached, the whole state is not invariant for permutation, and has a certain asymmetry for permutation.
  arXiv  Detail & Related papers  (2023-11-17T03:33:40Z)
• Lie Point Symmetry and Physics Informed Networks [59.56218517113066]
  We propose a loss function that informs the network about Lie point symmetries in the same way that PINN models try to enforce the underlying PDE through a loss function. Our symmetry loss ensures that the infinitesimal generators of the Lie group conserve the PDE solutions. Empirical evaluations indicate that the inductive bias introduced by the Lie point symmetries of the PDEs greatly boosts the sample efficiency of PINNs.
  arXiv  Detail & Related papers  (2023-11-07T19:07:16Z)
• Learning Layer-wise Equivariances Automatically using Gradients [66.81218780702125]
  Convolutions encode equivariance symmetries into neural networks leading to better generalisation performance. symmetries provide fixed hard constraints on the functions a network can represent, need to be specified in advance, and can not be adapted. Our goal is to allow flexible symmetry constraints that can automatically be learned from data using gradients.
  arXiv  Detail & Related papers  (2023-10-09T20:22:43Z)
• Symmetry Induces Structure and Constraint of Learning [0.0]
  We unveil the importance of the loss function symmetries in affecting, if not deciding, the learning behavior of machine learning models. Common instances of mirror symmetries in deep learning include rescaling, rotation, and permutation symmetry. We show that the theoretical framework can explain intriguing phenomena, such as the loss of plasticity and various collapse phenomena in neural networks.
  arXiv  Detail & Related papers  (2023-09-29T02:21:31Z)
• Identifying the Group-Theoretic Structure of Machine-Learned Symmetries [41.56233403862961]
  We propose methods for examining and identifying the group-theoretic structure of such machine-learned symmetries. As an application to particle physics, we demonstrate the identification of the residual symmetries after the spontaneous breaking of non-Abelian gauge symmetries.
  arXiv  Detail & Related papers  (2023-09-14T17:03:50Z)
• Symmetry protected entanglement in random mixed states [0.0]
  We study the effect of symmetry on tripartite entanglement properties of typical states in symmetric sectors of Hilbert space. In particular, we consider Abelian symmetries and derive an explicit expression for the logarithmic entanglement negativity of systems with $mathbbZ_N$ and $U(1)$ symmetry groups.
  arXiv  Detail & Related papers  (2021-11-30T19:00:07Z)
• Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning [7.310043452300738]
  In nature, symmetry governs regularities, while symmetry breaking brings texture. Recent experiments suggest that the symmetry of the loss function is closely related to the learning performance. We pose symmetry breaking as a new design principle by considering the symmetry of the learning rule in addition to the loss function.
  arXiv  Detail & Related papers  (2021-05-06T14:36:10Z)
• Symmetry Breaking in Symmetric Tensor Decomposition [44.181747424363245]
  We consider the nonsymmetry problem associated with computing the points rank decomposition of symmetric tensors. We show that critical points the loss function is detected by standard methods.
  arXiv  Detail & Related papers  (2021-03-10T18:11:22Z)
• Finding Symmetry Breaking Order Parameters with Euclidean Neural Networks [2.735801286587347]
  We demonstrate that symmetry equivariant neural networks uphold Curie's principle and can be used to articulate many symmetry-relevant scientific questions into simple optimization problems. We prove these properties mathematically and demonstrate them numerically by training a Euclidean symmetry equivariant neural network to learn symmetry-breaking input to deform a square into a rectangle and to generate octahedra tilting patterns in perovskites.
  arXiv  Detail & Related papers  (2020-07-04T17:24:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.