Deep Neural Networks with Efficient Guaranteed Invariances

• URL: http://arxiv.org/abs/2303.01567v1
• Date: Thu, 2 Mar 2023 20:44:45 GMT
• Title: Deep Neural Networks with Efficient Guaranteed Invariances
• Authors: Matthias Rath, Alexandru Paul Condurache
• Abstract summary: We address the problem of improving the performance and in particular the sample complexity of deep neural networks. Group-equivariant convolutions are a popular approach to obtain equivariant representations. We propose a multi-stream architecture, where each stream is invariant to a different transformation.
• Score: 77.99182201815763
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We address the problem of improving the performance and in particular the sample complexity of deep neural networks by enforcing and guaranteeing invariances to symmetry transformations rather than learning them from data. Group-equivariant convolutions are a popular approach to obtain equivariant representations. The desired corresponding invariance is then imposed using pooling operations. For rotations, it has been shown that using invariant integration instead of pooling further improves the sample complexity. In this contribution, we first expand invariant integration beyond rotations to flips and scale transformations. We then address the problem of incorporating multiple desired invariances into a single network. For this purpose, we propose a multi-stream architecture, where each stream is invariant to a different transformation such that the network can simultaneously benefit from multiple invariances. We demonstrate our approach with successful experiments on Scaled-MNIST, SVHN, CIFAR-10 and STL-10.

Related papers

• Multiset Transformer: Advancing Representation Learning in Persistence Diagrams [11.512742322405906]
  Multiset Transformer is a neural network that utilizes attention mechanisms specifically designed for multisets as inputs. The architecture integrates multiset-enhanced attentions with a pool-decomposition scheme, allowing multiplicities to be preserved across equivariant layers. Experimental results demonstrate that the Multiset Transformer outperforms existing neural network methods in the realm of persistence diagram representation learning.
  arXiv  Detail & Related papers  (2024-11-22T01:38:47Z)
• SO(2) and O(2) Equivariance in Image Recognition with Bessel-Convolutional Neural Networks [63.24965775030674]
  This work presents the development of Bessel-convolutional neural networks (B-CNNs) B-CNNs exploit a particular decomposition based on Bessel functions to modify the key operation between images and filters. Study is carried out to assess the performances of B-CNNs compared to other methods.
  arXiv  Detail & Related papers  (2023-04-18T18:06:35Z)
• Improving the Sample-Complexity of Deep Classification Networks with Invariant Integration [77.99182201815763]
  Leveraging prior knowledge on intraclass variance due to transformations is a powerful method to improve the sample complexity of deep neural networks. We propose a novel monomial selection algorithm based on pruning methods to allow an application to more complex problems. We demonstrate the improved sample complexity on the Rotated-MNIST, SVHN and CIFAR-10 datasets.
  arXiv  Detail & Related papers  (2022-02-08T16:16:11Z)
• Revisiting Transformation Invariant Geometric Deep Learning: Are Initial Representations All You Need? [80.86819657126041]
  We show that transformation-invariant and distance-preserving initial representations are sufficient to achieve transformation invariance. Specifically, we realize transformation-invariant and distance-preserving initial point representations by modifying multi-dimensional scaling. We prove that TinvNN can strictly guarantee transformation invariance, being general and flexible enough to be combined with the existing neural networks.
  arXiv  Detail & Related papers  (2021-12-23T03:52:33Z)
• Implicit Equivariance in Convolutional Networks [1.911678487931003]
  Implicitly Equivariant Networks (IEN) induce equivariant in the different layers of a standard CNN model. We show IEN outperforms the state-of-the-art rotation equivariant tracking method while providing faster inference speed.
  arXiv  Detail & Related papers  (2021-11-28T14:44:17Z)
• Frame Averaging for Invariant and Equivariant Network Design [50.87023773850824]
  We introduce Frame Averaging (FA), a framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types. We show that FA-based models have maximal expressive power in a broad setting. We propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs.
  arXiv  Detail & Related papers  (2021-10-07T11:05:23Z)
• Training or Architecture? How to Incorporate Invariance in Neural Networks [14.162739081163444]
  We propose a method for provably invariant network architectures with respect to group actions. In a nutshell, we intend to 'undo' any possible transformation before feeding the data into the actual network. We analyze properties of such approaches, extend them to equivariant networks, and demonstrate their advantages in terms of robustness as well as computational efficiency in several numerical examples.
  arXiv  Detail & Related papers  (2021-06-18T10:31:00Z)
• Group Equivariant Subsampling [60.53371517247382]
  Subsampling is used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions. We first introduce translation equivariant subsampling/upsampling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsampling/upsampling.
  arXiv  Detail & Related papers  (2021-06-10T16:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.