Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning

• URL: http://arxiv.org/abs/2105.02716v1
• Date: Thu, 6 May 2021 14:36:10 GMT
• Title: Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning
• Authors: Hidenori Tanaka, Daniel Kunin
• Abstract summary: In nature, symmetry governs regularities, while symmetry breaking brings texture. Recent experiments suggest that the symmetry of the loss function is closely related to the learning performance. We pose symmetry breaking as a new design principle by considering the symmetry of the learning rule in addition to the loss function.
• Score: 7.310043452300738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In nature, symmetry governs regularities, while symmetry breaking brings texture. Here, we reveal a novel role of symmetry breaking behind efficiency and stability in learning, a critical issue in machine learning. Recent experiments suggest that the symmetry of the loss function is closely related to the learning performance. This raises a fundamental question. Is such symmetry beneficial, harmful, or irrelevant to the success of learning? Here, we demystify this question and pose symmetry breaking as a new design principle by considering the symmetry of the learning rule in addition to the loss function. We model the discrete learning dynamics using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. We identify kinetic asymmetry unique to learning systems, where the kinetic energy often does not have the same symmetry as the potential (loss) function reflecting the non-physical symmetries of the loss function and the non-Euclidean metric used in learning rules. We generalize Noether's theorem known in physics to explicitly take into account this kinetic asymmetry and derive the resulting motion of the Noether charge. Finally, we apply our theory to modern deep networks with normalization layers and reveal a mechanism of implicit adaptive optimization induced by the kinetic symmetry breaking.

Related papers

• TANGO: Time-Reversal Latent GraphODE for Multi-Agent Dynamical Systems [43.39754726042369]
  We propose a simple-yet-effective self-supervised regularization term as a soft constraint that aligns the forward and backward trajectories predicted by a continuous graph neural network-based ordinary differential equation (GraphODE) It effectively imposes time-reversal symmetry to enable more accurate model predictions across a wider range of dynamical systems under classical mechanics. Experimental results on a variety of physical systems demonstrate the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2023-10-10T08:52:16Z)
• Learning Layer-wise Equivariances Automatically using Gradients [66.81218780702125]
  Convolutions encode equivariance symmetries into neural networks leading to better generalisation performance. symmetries provide fixed hard constraints on the functions a network can represent, need to be specified in advance, and can not be adapted. Our goal is to allow flexible symmetry constraints that can automatically be learned from data using gradients.
  arXiv  Detail & Related papers  (2023-10-09T20:22:43Z)
• Symmetry Induces Structure and Constraint of Learning [0.0]
  We unveil the importance of the loss function symmetries in affecting, if not deciding, the learning behavior of machine learning models. Common instances of mirror symmetries in deep learning include rescaling, rotation, and permutation symmetry. We show that the theoretical framework can explain intriguing phenomena, such as the loss of plasticity and various collapse phenomena in neural networks.
  arXiv  Detail & Related papers  (2023-09-29T02:21:31Z)
• Identifying the Group-Theoretic Structure of Machine-Learned Symmetries [41.56233403862961]
  We propose methods for examining and identifying the group-theoretic structure of such machine-learned symmetries. As an application to particle physics, we demonstrate the identification of the residual symmetries after the spontaneous breaking of non-Abelian gauge symmetries.
  arXiv  Detail & Related papers  (2023-09-14T17:03:50Z)
• Entanglement asymmetry as a probe of symmetry breaking [0.0]
  In extended quantum systems, quantifying how much a symmetry is broken is intrinsically bound to the subsystem of interest. We introduce a subsystem measure of symmetry breaking that we dub entanglement asymmetry. We find, expectedly, that larger is the subsystem, slower is the restoration, but also the counterintuitive result that more the symmetry is initially broken, faster it is restored.
  arXiv  Detail & Related papers  (2022-07-29T14:03:30Z)
• Exact solutions of interacting dissipative systems via weak symmetries [77.34726150561087]
  We analytically diagonalize the Liouvillian of a class Markovian dissipative systems with arbitrary strong interactions or nonlinearity. This enables an exact description of the full dynamics and dissipative spectrum. Our method is applicable to a variety of other systems, and could provide a powerful new tool for the study of complex driven-dissipative quantum systems.
  arXiv  Detail & Related papers  (2021-09-27T17:45:42Z)
• Noether: The More Things Change, the More Stay the Same [1.14219428942199]
  Noether's celebrated theorem states that symmetry leads to conserved quantities. In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. Symmetry can be thought of as one further important tool in understanding the performance of neural networks under gradient descent.
  arXiv  Detail & Related papers  (2021-04-12T14:41:05Z)
• Symmetry Breaking in Symmetric Tensor Decomposition [44.181747424363245]
  We consider the nonsymmetry problem associated with computing the points rank decomposition of symmetric tensors. We show that critical points the loss function is detected by standard methods.
  arXiv  Detail & Related papers  (2021-03-10T18:11:22Z)
• A Symmetric Loss Perspective of Reliable Machine Learning [87.68601212686086]
  We review how a symmetric loss can yield robust classification from corrupted labels in balanced error rate (BER) minimization. We demonstrate how the robust AUC method can benefit natural language processing in the problem where we want to learn only from relevant keywords.
  arXiv  Detail & Related papers  (2021-01-05T06:25:47Z)
• Latent symmetry induced degeneracies [0.0]
  We develop an approach to explain degeneracies by tracing them back to symmetries of an isospectral effective Hamiltonian. As an application, we relate the degeneracies induced by the rotation symmetry of a real Hamiltonian to a non-abelian latent symmetry group.
  arXiv  Detail & Related papers  (2020-11-26T17:37:30Z)
• Finding Symmetry Breaking Order Parameters with Euclidean Neural Networks [2.735801286587347]
  We demonstrate that symmetry equivariant neural networks uphold Curie's principle and can be used to articulate many symmetry-relevant scientific questions into simple optimization problems. We prove these properties mathematically and demonstrate them numerically by training a Euclidean symmetry equivariant neural network to learn symmetry-breaking input to deform a square into a rectangle and to generate octahedra tilting patterns in perovskites.
  arXiv  Detail & Related papers  (2020-07-04T17:24:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.