Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks

• URL: http://arxiv.org/abs/2312.08550v3
• Date: Fri, 14 Jun 2024 07:03:08 GMT
• Title: Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks
• Authors: Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, Sophia Sanborn,
• Abstract summary: We prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features -- a ubiquitous phenomenon in both biological and artificial learning systems.
• Score: 14.259918357897408
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we formally prove that, under certain conditions, if a neural network is invariant to a finite group then its weights recover the Fourier transform on that group. This provides a mathematical explanation for the emergence of Fourier features -- a ubiquitous phenomenon in both biological and artificial learning systems. The results hold even for non-commutative groups, in which case the Fourier transform encodes all the irreducible unitary group representations. Our findings have consequences for the problem of symmetry discovery. Specifically, we demonstrate that the algebraic structure of an unknown group can be recovered from the weights of a network that is at least approximately invariant within certain bounds. Overall, this work contributes to a foundation for an algebraic learning theory of invariant neural network representations.

Related papers

• Unsupervised Learning of Invariance Transformations [105.54048699217668]
  We develop an algorithmic framework for finding approximate graph automorphisms. We discuss how this framework can be used to find approximate automorphisms in weighted graphs in general.
  arXiv  Detail & Related papers  (2023-07-24T17:03:28Z)
• A tradeoff between universality of equivariant models and learnability of symmetries [0.0]
  We prove that it is impossible to simultaneously learn symmetries and functions equivariant under certain conditions. We analyze certain families of neural networks for whether they satisfy the conditions of the impossibility result. On the practical side, our analysis of group-convolutional neural networks allows us generalize the well-known convolution is all you need'' to non-homogeneous spaces.
  arXiv  Detail & Related papers  (2022-10-17T21:23:22Z)
• Equivariant Transduction through Invariant Alignment [71.45263447328374]
  We introduce a novel group-equivariant architecture that incorporates a group-in hard alignment mechanism. We find that our network's structure allows it to develop stronger equivariant properties than existing group-equivariant approaches. We additionally find that it outperforms previous group-equivariant networks empirically on the SCAN task.
  arXiv  Detail & Related papers  (2022-09-22T11:19:45Z)
• Bispectral Neural Networks [1.0323063834827415]
  We present a neural network architecture, Bispectral Neural Networks (BNNs) BNNs are able to simultaneously learn groups, their irreducible representations, and corresponding equivariant and complete-invariant maps.
  arXiv  Detail & Related papers  (2022-09-07T18:34:48Z)
• Equivariant Disentangled Transformation for Domain Generalization under Combination Shift [91.38796390449504]
  Combinations of domains and labels are not observed during training but appear in the test environment. We provide a unique formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement.
  arXiv  Detail & Related papers  (2022-08-03T12:31:31Z)
• Unified Fourier-based Kernel and Nonlinearity Design for Equivariant Networks on Homogeneous Spaces [52.424621227687894]
  We introduce a unified framework for group equivariant networks on homogeneous spaces. We take advantage of the sparsity of Fourier coefficients of the lifted feature fields. We show that other methods treating features as the Fourier coefficients in the stabilizer subgroup are special cases of our activation.
  arXiv  Detail & Related papers  (2022-06-16T17:59:01Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Lorentz Group Equivariant Neural Network for Particle Physics [58.56031187968692]
  We present a neural network architecture that is fully equivariant with respect to transformations under the Lorentz group. For classification tasks in particle physics, we demonstrate that such an equivariant architecture leads to drastically simpler models that have relatively few learnable parameters.
  arXiv  Detail & Related papers  (2020-06-08T17:54:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.