Regularizing Towards Permutation Invariance in Recurrent Models

• URL: http://arxiv.org/abs/2010.13055v1
• Date: Sun, 25 Oct 2020 07:46:51 GMT
• Title: Regularizing Towards Permutation Invariance in Recurrent Models
• Authors: Edo Cohen-Karlik, Avichai Ben David and Amir Globerson
• Abstract summary: We show that RNNs can be regularized towards permutation invariance, and that this can result in compact models. Existing solutions mostly suggest restricting the learning problem to hypothesis classes which are permutation invariant by design. We show that our method outperforms other permutation invariant approaches on synthetic and real world datasets.
• Score: 26.36835670113303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many machine learning problems the output should not depend on the order of the input. Such "permutation invariant" functions have been studied extensively recently. Here we argue that temporal architectures such as RNNs are highly relevant for such problems, despite the inherent dependence of RNNs on order. We show that RNNs can be regularized towards permutation invariance, and that this can result in compact models, as compared to non-recurrent architectures. We implement this idea via a novel form of stochastic regularization. Existing solutions mostly suggest restricting the learning problem to hypothesis classes which are permutation invariant by design. Our approach of enforcing permutation invariance via regularization gives rise to models which are \textit{semi permutation invariant} (e.g. invariant to some permutations and not to others). We show that our method outperforms other permutation invariant approaches on synthetic and real world datasets.

Related papers

• Permutation invariant functions: statistical tests, density estimation, and computationally efficient embedding [1.4316259003164373]
  Permutation invariance is among the most common symmetry that can be exploited to simplify complex problems in machine learning (ML) In this paper, we take a step back and examine these questions in several fundamental problems. Our methods for (i) and (iv) are based on a sorting trick and (ii) is based on an averaging trick.
  arXiv  Detail & Related papers  (2024-03-04T01:49:23Z)
• Probabilistic Invariant Learning with Randomized Linear Classifiers [24.485477981244593]
  We show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, we propose a class of binary classification models called Randomized Linears (RLCs)
  arXiv  Detail & Related papers  (2023-08-08T17:18:04Z)
• Deep Neural Networks with Efficient Guaranteed Invariances [77.99182201815763]
  We address the problem of improving the performance and in particular the sample complexity of deep neural networks. Group-equivariant convolutions are a popular approach to obtain equivariant representations. We propose a multi-stream architecture, where each stream is invariant to a different transformation.
  arXiv  Detail & Related papers  (2023-03-02T20:44:45Z)
• Equivariant Disentangled Transformation for Domain Generalization under Combination Shift [91.38796390449504]
  Combinations of domains and labels are not observed during training but appear in the test environment. We provide a unique formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement.
  arXiv  Detail & Related papers  (2022-08-03T12:31:31Z)
• Low Dimensional Invariant Embeddings for Universal Geometric Learning [6.405957390409045]
  This paper studies separating invariants: mappings on $D$ dimensional domains which are invariant to an appropriate group action, and which separate orbits. The motivation for this study comes from the usefulness of separating invariants in proving universality of equivariant neural network architectures.
  arXiv  Detail & Related papers  (2022-05-05T22:56:19Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Improving the Sample-Complexity of Deep Classification Networks with Invariant Integration [77.99182201815763]
  Leveraging prior knowledge on intraclass variance due to transformations is a powerful method to improve the sample complexity of deep neural networks. We propose a novel monomial selection algorithm based on pruning methods to allow an application to more complex problems. We demonstrate the improved sample complexity on the Rotated-MNIST, SVHN and CIFAR-10 datasets.
  arXiv  Detail & Related papers  (2022-02-08T16:16:11Z)
• Scalable Normalizing Flows for Permutation Invariant Densities [0.0]
  A promising approach defines a family of permutation invariant densities with continuous normalizing flows. We demonstrate how calculating the trace, a crucial step in this method, raises issues that occur both during training and inference. We propose an alternative way of defining permutation equivariant transformations that give closed form trace.
  arXiv  Detail & Related papers  (2020-10-07T07:51:30Z)
• Permutation Invariant Graph Generation via Score-Based Generative Modeling [114.12935776726606]
  We propose a permutation invariant approach to modeling graphs, using the recent framework of score-based generative modeling. In particular, we design a permutation equivariant, multi-channel graph neural network to model the gradient of the data distribution at the input graph. For graph generation, we find that our learning approach achieves better or comparable results to existing models on benchmark datasets.
  arXiv  Detail & Related papers  (2020-03-02T03:06:14Z)
• A Permutation-Equivariant Neural Network Architecture For Auction Design [49.41561446069114]
  Design of an incentive compatible auction that maximizes expected revenue is a central problem in Auction Design. In this work, we consider auction design problems that have permutationequivariant symmetry and construct a neural architecture that is capable of perfectly recovering the permutationequi optimal mechanism.
  arXiv  Detail & Related papers  (2020-03-02T00:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.