LieTransformer: Equivariant self-attention for Lie Groups
- URL: http://arxiv.org/abs/2012.10885v3
- Date: Thu, 8 Apr 2021 16:19:14 GMT
- Title: LieTransformer: Equivariant self-attention for Lie Groups
- Authors: Michael Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont,
Yee Whye Teh, Hyunjik Kim
- Abstract summary: Group equivariant neural networks are used as building blocks of group invariant neural networks.
We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models.
We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
- Score: 49.9625160479096
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Group equivariant neural networks are used as building blocks of group
invariant neural networks, which have been shown to improve generalisation
performance and data efficiency through principled parameter sharing. Such
works have mostly focused on group equivariant convolutions, building on the
result that group equivariant linear maps are necessarily convolutions. In this
work, we extend the scope of the literature to self-attention, that is emerging
as a prominent building block of deep learning models. We propose the
LieTransformer, an architecture composed of LieSelfAttention layers that are
equivariant to arbitrary Lie groups and their discrete subgroups. We
demonstrate the generality of our approach by showing experimental results that
are competitive to baseline methods on a wide range of tasks: shape counting on
point clouds, molecular property regression and modelling particle trajectories
under Hamiltonian dynamics.
Related papers
- Lie Group Decompositions for Equivariant Neural Networks [12.139222986297261]
We show how convolution kernels can be parametrized to build models equivariant with respect to affine transformations.
We evaluate the robustness and out-of-distribution generalisation capability of our model on the benchmark affine-invariant classification task.
arXiv Detail & Related papers (2023-10-17T16:04:33Z) - A General Framework for Equivariant Neural Networks on Reductive Lie
Groups [2.0769531810371307]
Reductive Lie Groups play essential roles across scientific fields as diverse as high energy physics, quantum mechanics, quantum chromodynamics, molecular dynamics, computer vision, and imaging.
We present a general Equivariant Neural Network architecture capable of respecting the finite-dimensional representations of any reductive Lie Group G.
arXiv Detail & Related papers (2023-05-31T18:09:37Z) - Unified Fourier-based Kernel and Nonlinearity Design for Equivariant
Networks on Homogeneous Spaces [52.424621227687894]
We introduce a unified framework for group equivariant networks on homogeneous spaces.
We take advantage of the sparsity of Fourier coefficients of the lifted feature fields.
We show that other methods treating features as the Fourier coefficients in the stabilizer subgroup are special cases of our activation.
arXiv Detail & Related papers (2022-06-16T17:59:01Z) - Topographic VAEs learn Equivariant Capsules [84.33745072274942]
We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables.
We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST.
We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
arXiv Detail & Related papers (2021-09-03T09:25:57Z) - Commutative Lie Group VAE for Disentanglement Learning [96.32813624341833]
We view disentanglement learning as discovering an underlying structure that equivariantly reflects the factorized variations shown in data.
A simple model named Commutative Lie Group VAE is introduced to realize the group-based disentanglement learning.
Experiments show that our model can effectively learn disentangled representations without supervision, and can achieve state-of-the-art performance without extra constraints.
arXiv Detail & Related papers (2021-06-07T07:03:14Z) - A Practical Method for Constructing Equivariant Multilayer Perceptrons
for Arbitrary Matrix Groups [115.58550697886987]
We provide a completely general algorithm for solving for the equivariant layers of matrix groups.
In addition to recovering solutions from other works as special cases, we construct multilayer perceptrons equivariant to multiple groups that have never been tackled before.
Our approach outperforms non-equivariant baselines, with applications to particle physics and dynamical systems.
arXiv Detail & Related papers (2021-04-19T17:21:54Z) - Group Equivariant Conditional Neural Processes [30.134634059773703]
We present the group equivariant conditional neural process (EquivCNP)
We show that EquivCNP achieves comparable performance to conventional conditional neural processes in a 1D regression task.
arXiv Detail & Related papers (2021-02-17T13:50:07Z) - Learning Irreducible Representations of Noncommutative Lie Groups [3.1727619150610837]
Recent work has constructed neural networks that are equivariant to continuous symmetry groups such as 2D and 3D rotations.
We present two contributions motivated by frontier applications of equivariance beyond rotations and translations.
arXiv Detail & Related papers (2020-06-01T05:14:29Z) - Generalizing Convolutional Neural Networks for Equivariance to Lie
Groups on Arbitrary Continuous Data [52.78581260260455]
We propose a general method to construct a convolutional layer that is equivariant to transformations from any specified Lie group.
We apply the same model architecture to images, ball-and-stick molecular data, and Hamiltonian dynamical systems.
arXiv Detail & Related papers (2020-02-25T17:40:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.