Unsupervised Learning of Group Invariant and Equivariant Representations

• URL: http://arxiv.org/abs/2202.07559v3
• Date: Fri, 12 Apr 2024 13:16:54 GMT
• Title: Unsupervised Learning of Group Invariant and Equivariant Representations
• Authors: Robin Winter, Marco Bertolini, Tuan Le, Frank Noé, Djork-Arné Clevert,
• Abstract summary: We extend group invariant and equivariant representation learning to the field of unsupervised deep learning. We propose a general learning strategy based on an encoder-decoder framework in which the latent representation is separated in an invariant term and an equivariant group action component. The key idea is that the network learns to encode and decode data to and from a group-invariant representation by additionally learning to predict the appropriate group action to align input and output pose to solve the reconstruction task.
• Score: 10.252723257176566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Equivariant neural networks, whose hidden features transform according to representations of a group G acting on the data, exhibit training efficiency and an improved generalisation performance. In this work, we extend group invariant and equivariant representation learning to the field of unsupervised deep learning. We propose a general learning strategy based on an encoder-decoder framework in which the latent representation is separated in an invariant term and an equivariant group action component. The key idea is that the network learns to encode and decode data to and from a group-invariant representation by additionally learning to predict the appropriate group action to align input and output pose to solve the reconstruction task. We derive the necessary conditions on the equivariant encoder, and we present a construction valid for any G, both discrete and continuous. We describe explicitly our construction for rotations, translations and permutations. We test the validity and the robustness of our approach in a variety of experiments with diverse data types employing different network architectures.

Related papers

• Learning Rotation-Equivariant Features for Visual Correspondence [41.79256655501003]
  We introduce a self-supervised learning framework to extract discriminative rotation-invariant descriptors. Thanks to employing group-equivariant CNNs, our method effectively learns to obtain rotation-equivariant features and their orientations explicitly. Our method demonstrates state-of-the-art matching accuracy among existing rotation-invariant descriptors under varying rotation.
  arXiv  Detail & Related papers  (2023-03-25T13:42:07Z)
• Self-Supervised Learning for Group Equivariant Neural Networks [75.62232699377877]
  Group equivariant neural networks are the models whose structure is restricted to commute with the transformations on the input. We propose two concepts for self-supervised tasks: equivariant pretext labels and invariant contrastive loss. Experiments on standard image recognition benchmarks demonstrate that the equivariant neural networks exploit the proposed self-supervised tasks.
  arXiv  Detail & Related papers  (2023-03-08T08:11:26Z)
• Deep Neural Networks with Efficient Guaranteed Invariances [77.99182201815763]
  We address the problem of improving the performance and in particular the sample complexity of deep neural networks. Group-equivariant convolutions are a popular approach to obtain equivariant representations. We propose a multi-stream architecture, where each stream is invariant to a different transformation.
  arXiv  Detail & Related papers  (2023-03-02T20:44:45Z)
• Equivariant Transduction through Invariant Alignment [71.45263447328374]
  We introduce a novel group-equivariant architecture that incorporates a group-in hard alignment mechanism. We find that our network's structure allows it to develop stronger equivariant properties than existing group-equivariant approaches. We additionally find that it outperforms previous group-equivariant networks empirically on the SCAN task.
  arXiv  Detail & Related papers  (2022-09-22T11:19:45Z)
• Bispectral Neural Networks [1.0323063834827415]
  We present a neural network architecture, Bispectral Neural Networks (BNNs) BNNs are able to simultaneously learn groups, their irreducible representations, and corresponding equivariant and complete-invariant maps.
  arXiv  Detail & Related papers  (2022-09-07T18:34:48Z)
• Transformation Coding: Simple Objectives for Equivariant Representations [17.544323284367927]
  We present a non-generative approach to deep representation learning that seeks equivariant deep embedding through simple objectives. In contrast to existing equivariant networks, our transformation coding approach does not constrain the choice of the feed-forward layer or the architecture.
  arXiv  Detail & Related papers  (2022-02-19T01:43:13Z)
• Topographic VAEs learn Equivariant Capsules [84.33745072274942]
  We introduce the Topographic VAE: a novel method for efficiently training deep generative models with topographically organized latent variables. We show that such a model indeed learns to organize its activations according to salient characteristics such as digit class, width, and style on MNIST. We demonstrate approximate equivariance to complex transformations, expanding upon the capabilities of existing group equivariant neural networks.
  arXiv  Detail & Related papers  (2021-09-03T09:25:57Z)
• Group Equivariant Subsampling [60.53371517247382]
  Subsampling is used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions. We first introduce translation equivariant subsampling/upsampling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsampling/upsampling.
  arXiv  Detail & Related papers  (2021-06-10T16:14:00Z)
• Group Equivariant Neural Architecture Search via Group Decomposition and Reinforcement Learning [17.291131923335918]
  We prove a new group-theoretic result in the context of equivariant neural networks. We also design an algorithm to construct equivariant networks that significantly improves computational complexity. We use deep Q-learning to search for group equivariant networks that maximize performance.
  arXiv  Detail & Related papers  (2021-04-10T19:37:25Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.