Investigating how ReLU-networks encode symmetries

• URL: http://arxiv.org/abs/2305.17017v2
• Date: Fri, 8 Dec 2023 18:27:31 GMT
• Title: Investigating how ReLU-networks encode symmetries
• Authors: Georg B\"okman and Fredrik Kahl
• Abstract summary: We investigate whether equivariance of a network implies that all layers are equivariant. We conjecture that CNNs trained to be equivariant will exhibit layerwise equivariance. We show that it is typically easier to merge a network with a group-transformed version of itself than merging two different networks.
• Score: 13.935148870831396
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Many data symmetries can be described in terms of group equivariance and the most common way of encoding group equivariances in neural networks is by building linear layers that are group equivariant. In this work we investigate whether equivariance of a network implies that all layers are equivariant. On the theoretical side we find cases where equivariance implies layerwise equivariance, but also demonstrate that this is not the case generally. Nevertheless, we conjecture that CNNs that are trained to be equivariant will exhibit layerwise equivariance and explain how this conjecture is a weaker version of the recent permutation conjecture by Entezari et al. [2022]. We perform quantitative experiments with VGG-nets on CIFAR10 and qualitative experiments with ResNets on ImageNet to illustrate and support our theoretical findings. These experiments are not only of interest for understanding how group equivariance is encoded in ReLU-networks, but they also give a new perspective on Entezari et al.'s permutation conjecture as we find that it is typically easier to merge a network with a group-transformed version of itself than merging two different networks.

Related papers

• A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs [5.141137421503899]
  Steerable convolutional neural networks (SCNNs) enhance task performance by modelling geometric symmetries. Yet, unknown or varying symmetries can lead to overconstrained weights and decreased performance. This paper introduces a probabilistic method to learn the degree of equivariance in SCNNs.
  arXiv  Detail & Related papers  (2024-06-06T10:45:19Z)
• Using and Abusing Equivariance [10.70891251559827]
  We show how Group Equivariant Convolutional Neural Networks use subsampling to learn to break equivariance to their symmetries. We show that a change in the input dimension of a network as small as a single pixel can be enough for commonly used architectures to become approximately equivariant, rather than exactly.
  arXiv  Detail & Related papers  (2023-08-22T09:49:26Z)
• Self-Supervised Learning for Group Equivariant Neural Networks [75.62232699377877]
  Group equivariant neural networks are the models whose structure is restricted to commute with the transformations on the input. We propose two concepts for self-supervised tasks: equivariant pretext labels and invariant contrastive loss. Experiments on standard image recognition benchmarks demonstrate that the equivariant neural networks exploit the proposed self-supervised tasks.
  arXiv  Detail & Related papers  (2023-03-08T08:11:26Z)
• Deep Neural Networks with Efficient Guaranteed Invariances [77.99182201815763]
  We address the problem of improving the performance and in particular the sample complexity of deep neural networks. Group-equivariant convolutions are a popular approach to obtain equivariant representations. We propose a multi-stream architecture, where each stream is invariant to a different transformation.
  arXiv  Detail & Related papers  (2023-03-02T20:44:45Z)
• Learning Invariant Representations for Equivariant Neural Networks Using Orthogonal Moments [9.680414207552722]
  The convolutional layers of standard convolutional neural networks (CNNs) are equivariant to translation. Recently, a new class of CNNs is proposed in which the conventional layers of CNNs are replaced with equivariant convolution, pooling, and batch-normalization layers.
  arXiv  Detail & Related papers  (2022-09-22T11:48:39Z)
• Equivariant Transduction through Invariant Alignment [71.45263447328374]
  We introduce a novel group-equivariant architecture that incorporates a group-in hard alignment mechanism. We find that our network's structure allows it to develop stronger equivariant properties than existing group-equivariant approaches. We additionally find that it outperforms previous group-equivariant networks empirically on the SCAN task.
  arXiv  Detail & Related papers  (2022-09-22T11:19:45Z)
• Equivariant Disentangled Transformation for Domain Generalization under Combination Shift [91.38796390449504]
  Combinations of domains and labels are not observed during training but appear in the test environment. We provide a unique formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement.
  arXiv  Detail & Related papers  (2022-08-03T12:31:31Z)
• Equivariance versus Augmentation for Spherical Images [0.7388859384645262]
  We analyze the role of rotational equivariance in convolutional neural networks (CNNs) applied to spherical images. We compare the performance of the group equivariant networks known as S2CNNs and standard non-equivariant CNNs trained with an increasing amount of data augmentation.
  arXiv  Detail & Related papers  (2022-02-08T16:49:30Z)
• Group Equivariant Subsampling [60.53371517247382]
  Subsampling is used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions. We first introduce translation equivariant subsampling/upsampling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsampling/upsampling.
  arXiv  Detail & Related papers  (2021-06-10T16:14:00Z)
• Group Equivariant Neural Architecture Search via Group Decomposition and Reinforcement Learning [17.291131923335918]
  We prove a new group-theoretic result in the context of equivariant neural networks. We also design an algorithm to construct equivariant networks that significantly improves computational complexity. We use deep Q-learning to search for group equivariant networks that maximize performance.
  arXiv  Detail & Related papers  (2021-04-10T19:37:25Z)
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data [52.78581260260455]
  We propose a general method to construct a convolutional layer that is equivariant to transformations from any specified Lie group. We apply the same model architecture to images, ball-and-stick molecular data, and Hamiltonian dynamical systems.
  arXiv  Detail & Related papers  (2020-02-25T17:40:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.