Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data

• URL: http://arxiv.org/abs/2002.12880v3
• Date: Thu, 24 Sep 2020 15:08:36 GMT
• Title: Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• Authors: Marc Finzi, Samuel Stanton, Pavel Izmailov, Andrew Gordon Wilson
• Abstract summary: We propose a general method to construct a convolutional layer that is equivariant to transformations from any specified Lie group. We apply the same model architecture to images, ball-and-stick molecular data, and Hamiltonian dynamical systems.
• Score: 52.78581260260455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The translation equivariance of convolutional layers enables convolutional neural networks to generalize well on image problems. While translation equivariance provides a powerful inductive bias for images, we often additionally desire equivariance to other transformations, such as rotations, especially for non-image data. We propose a general method to construct a convolutional layer that is equivariant to transformations from any specified Lie group with a surjective exponential map. Incorporating equivariance to a new group requires implementing only the group exponential and logarithm maps, enabling rapid prototyping. Showcasing the simplicity and generality of our method, we apply the same model architecture to images, ball-and-stick molecular data, and Hamiltonian dynamical systems. For Hamiltonian systems, the equivariance of our models is especially impactful, leading to exact conservation of linear and angular momentum.

Related papers

• The Lie Derivative for Measuring Learned Equivariance [84.29366874540217]
  We study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. We find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities. For example, transformers can be more equivariant than convolutional neural networks after training.
  arXiv  Detail & Related papers  (2022-10-06T15:20:55Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Beyond permutation equivariance in graph networks [1.713291434132985]
  We introduce a novel architecture for graph networks which is equivariant to the Euclidean group in $n$-dimensions. Our model is designed to work with graph networks in their most general form, thus including particular variants as special cases.
  arXiv  Detail & Related papers  (2021-03-25T18:36:09Z)
• Group Equivariant Conditional Neural Processes [30.134634059773703]
  We present the group equivariant conditional neural process (EquivCNP) We show that EquivCNP achieves comparable performance to conventional conditional neural processes in a 1D regression task.
  arXiv  Detail & Related papers  (2021-02-17T13:50:07Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)
• Learning Equivariant Representations [10.745691354609738]
  Convolutional neural networks (CNNs) are successful examples of this principle. We propose equivariant models for different transformations defined by groups of symmetries. These models leverage symmetries in the data to reduce sample and model complexity and improve generalization performance.
  arXiv  Detail & Related papers  (2020-12-04T18:46:17Z)
• The general theory of permutation equivarant neural networks and higher order graph variational encoders [6.117371161379209]
  We derive formulae for general permutation equivariant layers, including the case where the layer acts on matrices by permuting their rows and columns simultaneously. This case arises naturally in graph learning and relation learning applications. We present a second order graph variational encoder, and show that the latent distribution of equivariant generative models must be exchangeable.
  arXiv  Detail & Related papers  (2020-04-08T13:29:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.