Low Dimensional Invariant Embeddings for Universal Geometric Learning

• URL: http://arxiv.org/abs/2205.02956v3
• Date: Tue, 21 Nov 2023 15:57:17 GMT
• Title: Low Dimensional Invariant Embeddings for Universal Geometric Learning
• Authors: Nadav Dym and Steven J. Gortler
• Abstract summary: This paper studies separating invariants: mappings on $D$ dimensional domains which are invariant to an appropriate group action, and which separate orbits. The motivation for this study comes from the usefulness of separating invariants in proving universality of equivariant neural network architectures.
• Score: 6.405957390409045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper studies separating invariants: mappings on $D$ dimensional domains which are invariant to an appropriate group action, and which separate orbits. The motivation for this study comes from the usefulness of separating invariants in proving universality of equivariant neural network architectures. We observe that in several cases the cardinality of separating invariants proposed in the machine learning literature is much larger than the dimension $D$. As a result, the theoretical universal constructions based on these separating invariants is unrealistically large. Our goal in this paper is to resolve this issue. We show that when a continuous family of semi-algebraic separating invariants is available, separation can be obtained by randomly selecting $2D+1 $ of these invariants. We apply this methodology to obtain an efficient scheme for computing separating invariants for several classical group actions which have been studied in the invariant learning literature. Examples include matrix multiplication actions on point clouds by permutations, rotations, and various other linear groups. Often the requirement of invariant separation is relaxed and only generic separation is required. In this case, we show that only $D+1$ invariants are required. More importantly, generic invariants are often significantly easier to compute, as we illustrate by discussing generic and full separation for weighted graphs. Finally we outline an approach for proving that separating invariants can be constructed also when the random parameters have finite precision.

Related papers

• Global optimality under amenable symmetry constraints [0.5656581242851759]
  We show the interplay between convexity, the group, and the underlying vector space, which is typically infinite-dimensional. We apply this toolkit to the invariant optimality problem. It yields new results on invariant kernel mean embeddings and risk-optimal invariant couplings.
  arXiv  Detail & Related papers  (2024-02-12T12:38:20Z)
• Lie Group Decompositions for Equivariant Neural Networks [12.139222986297261]
  We show how convolution kernels can be parametrized to build models equivariant with respect to affine transformations. We evaluate the robustness and out-of-distribution generalisation capability of our model on the benchmark affine-invariant classification task.
  arXiv  Detail & Related papers  (2023-10-17T16:04:33Z)
• Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups [0.0]
  We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks.
  arXiv  Detail & Related papers  (2023-09-24T19:40:15Z)
• Deep Neural Networks with Efficient Guaranteed Invariances [77.99182201815763]
  We address the problem of improving the performance and in particular the sample complexity of deep neural networks. Group-equivariant convolutions are a popular approach to obtain equivariant representations. We propose a multi-stream architecture, where each stream is invariant to a different transformation.
  arXiv  Detail & Related papers  (2023-03-02T20:44:45Z)
• Machine learning and invariant theory [10.178220223515956]
  We introduce the topic and explain a couple of methods to explicitly parameterize equivariant functions. We explicate a general procedure, attributed to Malgrange, to express all maps between linear spaces that are equivariant under the action of a group $G$.
  arXiv  Detail & Related papers  (2022-09-29T17:52:17Z)
• Equivariant Disentangled Transformation for Domain Generalization under Combination Shift [91.38796390449504]
  Combinations of domains and labels are not observed during training but appear in the test environment. We provide a unique formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement.
  arXiv  Detail & Related papers  (2022-08-03T12:31:31Z)
• Measuring dissimilarity with diffeomorphism invariance [94.02751799024684]
  We introduce DID, a pairwise dissimilarity measure applicable to a wide range of data spaces. We prove that DID enjoys properties which make it relevant for theoretical study and practical use.
  arXiv  Detail & Related papers  (2022-02-11T13:51:30Z)
• Improving the Sample-Complexity of Deep Classification Networks with Invariant Integration [77.99182201815763]
  Leveraging prior knowledge on intraclass variance due to transformations is a powerful method to improve the sample complexity of deep neural networks. We propose a novel monomial selection algorithm based on pruning methods to allow an application to more complex problems. We demonstrate the improved sample complexity on the Rotated-MNIST, SVHN and CIFAR-10 datasets.
  arXiv  Detail & Related papers  (2022-02-08T16:16:11Z)
• Group Equivariant Subsampling [60.53371517247382]
  Subsampling is used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions. We first introduce translation equivariant subsampling/upsampling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsampling/upsampling.
  arXiv  Detail & Related papers  (2021-06-10T16:14:00Z)
• A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups [115.58550697886987]
  We provide a completely general algorithm for solving for the equivariant layers of matrix groups. In addition to recovering solutions from other works as special cases, we construct multilayer perceptrons equivariant to multiple groups that have never been tackled before. Our approach outperforms non-equivariant baselines, with applications to particle physics and dynamical systems.
  arXiv  Detail & Related papers  (2021-04-19T17:21:54Z)
• Regularizing Towards Permutation Invariance in Recurrent Models [26.36835670113303]
  We show that RNNs can be regularized towards permutation invariance, and that this can result in compact models. Existing solutions mostly suggest restricting the learning problem to hypothesis classes which are permutation invariant by design. We show that our method outperforms other permutation invariant approaches on synthetic and real world datasets.
  arXiv  Detail & Related papers  (2020-10-25T07:46:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.