Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups

• URL: http://arxiv.org/abs/2309.13736v2
• Date: Fri, 26 Jan 2024 13:13:40 GMT
• Title: Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups
• Authors: Kathl\'en Kohn, Anna-Laura Sattelberger, Vahid Shahverdi
• Abstract summary: We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The set of functions parameterized by a linear fully-connected neural network is a determinantal variety. We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. Examples of such group actions are translations or $90^\circ$ rotations on images. We describe such equivariant or invariant subvarieties as direct products of determinantal varieties, from which we deduce their dimension, degree, Euclidean distance degree, and their singularities. We fully characterize invariance for arbitrary permutation groups, and equivariance for cyclic groups. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks in terms of sparsity and weight-sharing properties. We prove that all invariant linear functions can be parameterized by a single linear autoencoder with a weight-sharing property imposed by the cycle decomposition of the considered permutation. The space of rank-bounded equivariant functions has several irreducible components, so it can {\em not} be parameterized by a single network -- but each irreducible component can. Finally, we show that minimizing the squared-error loss on our invariant or equivariant networks reduces to minimizing the Euclidean distance from determinantal varieties via the Eckart--Young theorem.

Related papers

• Learning functions on symmetric matrices and point clouds via lightweight invariant features [26.619014249559942]
  We present a formulation for machine learning of functions on symmetric matrices that are invariant with respect to the action of permutations. We show that these invariant features can separate all distinct orbits of symmetric matrices except for a measure zero set. For point clouds in a fixed dimension, we prove that the number of invariant features can be reduced, generically without losing expressivity.
  arXiv  Detail & Related papers  (2024-05-13T18:24:03Z)
• A Characterization Theorem for Equivariant Networks with Point-wise Activations [13.00676132572457]
  We prove that rotation-equivariant networks can only be invariant, as it happens for any network which is equivariant with respect to connected compact groups. We show that feature spaces of disentangled steerable convolutional neural networks are trivial representations.
  arXiv  Detail & Related papers  (2024-01-17T14:30:46Z)
• Self-Supervised Learning for Group Equivariant Neural Networks [75.62232699377877]
  Group equivariant neural networks are the models whose structure is restricted to commute with the transformations on the input. We propose two concepts for self-supervised tasks: equivariant pretext labels and invariant contrastive loss. Experiments on standard image recognition benchmarks demonstrate that the equivariant neural networks exploit the proposed self-supervised tasks.
  arXiv  Detail & Related papers  (2023-03-08T08:11:26Z)
• Equivariant Disentangled Transformation for Domain Generalization under Combination Shift [91.38796390449504]
  Combinations of domains and labels are not observed during training but appear in the test environment. We provide a unique formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement.
  arXiv  Detail & Related papers  (2022-08-03T12:31:31Z)
• Relaxing Equivariance Constraints with Non-stationary Continuous Filters [20.74154804898478]
  The proposed parameterization can be thought of as a building block to allow adjustable symmetry structure in neural networks. Compared to non-equivariant or strict-equivariant baselines, we experimentally verify that soft equivariance leads to improved performance in terms of test accuracy on CIFAR-10 and CIFAR-100 image classification tasks.
  arXiv  Detail & Related papers  (2022-04-14T18:08:36Z)
• Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views? [21.06669693699965]
  We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action. We show how this relation extends to operations such as convolutions, element-wise nonlinearities, and global and local pooling.
  arXiv  Detail & Related papers  (2021-10-14T15:46:53Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Commutative Lie Group VAE for Disentanglement Learning [96.32813624341833]
  We view disentanglement learning as discovering an underlying structure that equivariantly reflects the factorized variations shown in data. A simple model named Commutative Lie Group VAE is introduced to realize the group-based disentanglement learning. Experiments show that our model can effectively learn disentangled representations without supervision, and can achieve state-of-the-art performance without extra constraints.
  arXiv  Detail & Related papers  (2021-06-07T07:03:14Z)
• Group Equivariant Conditional Neural Processes [30.134634059773703]
  We present the group equivariant conditional neural process (EquivCNP) We show that EquivCNP achieves comparable performance to conventional conditional neural processes in a 1D regression task.
  arXiv  Detail & Related papers  (2021-02-17T13:50:07Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)
• Invariant Feature Coding using Tensor Product Representation [75.62232699377877]
  We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier. A novel feature model that explicitly consider group action is proposed for principal component analysis and k-means clustering.
  arXiv  Detail & Related papers  (2019-06-05T07:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.