Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?

• URL: http://arxiv.org/abs/2110.07472v1
• Date: Thu, 14 Oct 2021 15:46:53 GMT
• Title: Capacity of Group-invariant Linear Readouts from Equivariant Representations: How Many Objects can be Linearly Classified Under All Possible Views?
• Authors: Matthew Farrell, Blake Bordelon, Shubhendu Trivedi and Cengiz Pehlevan
• Abstract summary: We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action. We show how this relation extends to operations such as convolutions, element-wise nonlinearities, and global and local pooling.
• Score: 21.06669693699965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Equivariance has emerged as a desirable property of representations of objects subject to identity-preserving transformations that constitute a group, such as translations and rotations. However, the expressivity of a representation constrained by group equivariance is still not fully understood. We address this gap by providing a generalization of Cover's Function Counting Theorem that quantifies the number of linearly separable and group-invariant binary dichotomies that can be assigned to equivariant representations of objects. We find that the fraction of separable dichotomies is determined by the dimension of the space that is fixed by the group action. We show how this relation extends to operations such as convolutions, element-wise nonlinearities, and global and local pooling. While other operations do not change the fraction of separable dichotomies, local pooling decreases the fraction, despite being a highly nonlinear operation. Finally, we test our theory on intermediate representations of randomly initialized and fully trained convolutional neural networks and find perfect agreement.

Related papers

• Lie Group Decompositions for Equivariant Neural Networks [12.139222986297261]
  We show how convolution kernels can be parametrized to build models equivariant with respect to affine transformations. We evaluate the robustness and out-of-distribution generalisation capability of our model on the benchmark affine-invariant classification task.
  arXiv  Detail & Related papers  (2023-10-17T16:04:33Z)
• Algebras of actions in an agent's representations of the world [51.06229789727133]
  We use our framework to reproduce the symmetry-based representations from the symmetry-based disentangled representation learning formalism. We then study the algebras of the transformations of worlds with features that occur in simple reinforcement learning scenarios. Using computational methods, that we developed, we extract the algebras of the transformations of these worlds and classify them according to their properties.
  arXiv  Detail & Related papers  (2023-10-02T18:24:51Z)
• Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups [0.0]
  We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks.
  arXiv  Detail & Related papers  (2023-09-24T19:40:15Z)
• Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance [31.875925637190328]
  In this paper, we present Banana, a Banach fixed-point network for equivariant segmentation with inter-part equivariance by construction. Our key insight is to iteratively solve a fixed-point problem, where point-part assignment labels and per-part SE(3)-equivariance co-evolve simultaneously. Our formulation naturally provides a strict definition of inter-part equivariance that generalizes to unseen inter-part configurations.
  arXiv  Detail & Related papers  (2023-05-25T17:59:32Z)
• A tradeoff between universality of equivariant models and learnability of symmetries [0.0]
  We prove that it is impossible to simultaneously learn symmetries and functions equivariant under certain conditions. We analyze certain families of neural networks for whether they satisfy the conditions of the impossibility result. On the practical side, our analysis of group-convolutional neural networks allows us generalize the well-known convolution is all you need'' to non-homogeneous spaces.
  arXiv  Detail & Related papers  (2022-10-17T21:23:22Z)
• Equivariant Disentangled Transformation for Domain Generalization under Combination Shift [91.38796390449504]
  Combinations of domains and labels are not observed during training but appear in the test environment. We provide a unique formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement.
  arXiv  Detail & Related papers  (2022-08-03T12:31:31Z)
• Group Equivariant Subsampling [60.53371517247382]
  Subsampling is used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions. We first introduce translation equivariant subsampling/upsampling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsampling/upsampling.
  arXiv  Detail & Related papers  (2021-06-10T16:14:00Z)
• Commutative Lie Group VAE for Disentanglement Learning [96.32813624341833]
  We view disentanglement learning as discovering an underlying structure that equivariantly reflects the factorized variations shown in data. A simple model named Commutative Lie Group VAE is introduced to realize the group-based disentanglement learning. Experiments show that our model can effectively learn disentangled representations without supervision, and can achieve state-of-the-art performance without extra constraints.
  arXiv  Detail & Related papers  (2021-06-07T07:03:14Z)
• GroupifyVAE: from Group-based Definition to VAE-based Unsupervised Representation Disentanglement [91.9003001845855]
  VAE-based unsupervised disentanglement can not be achieved without introducing other inductive bias. We address VAE-based unsupervised disentanglement by leveraging the constraints derived from the Group Theory based definition as the non-probabilistic inductive bias. We train 1800 models covering the most prominent VAE-based models on five datasets to verify the effectiveness of our method.
  arXiv  Detail & Related papers  (2021-02-20T09:49:51Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)
• Learning Irreducible Representations of Noncommutative Lie Groups [3.1727619150610837]
  Recent work has constructed neural networks that are equivariant to continuous symmetry groups such as 2D and 3D rotations. We present two contributions motivated by frontier applications of equivariance beyond rotations and translations.
  arXiv  Detail & Related papers  (2020-06-01T05:14:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.