Scalable Normalizing Flows for Permutation Invariant Densities

• URL: http://arxiv.org/abs/2010.03242v2
• Date: Wed, 30 Jun 2021 10:50:02 GMT
• Title: Scalable Normalizing Flows for Permutation Invariant Densities
• Authors: Marin Bilo\v{s}, Stephan G\"unnemann
• Abstract summary: A promising approach defines a family of permutation invariant densities with continuous normalizing flows. We demonstrate how calculating the trace, a crucial step in this method, raises issues that occur both during training and inference. We propose an alternative way of defining permutation equivariant transformations that give closed form trace.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modeling sets is an important problem in machine learning since this type of data can be found in many domains. A promising approach defines a family of permutation invariant densities with continuous normalizing flows. This allows us to maximize the likelihood directly and sample new realizations with ease. In this work, we demonstrate how calculating the trace, a crucial step in this method, raises issues that occur both during training and inference, limiting its practicality. We propose an alternative way of defining permutation equivariant transformations that give closed form trace. This leads not only to improvements while training, but also to better final performance. We demonstrate the benefits of our approach on point processes and general set modeling.

Related papers

• Unnatural Algorithms in Machine Learning [0.0]
  We show that optimization algorithms with this property can be viewed as discrete approximations of natural gradient descent. We introduce a simple method of introducing this naturality more generally and examine a number of popular machine learning training algorithms.
  arXiv  Detail & Related papers  (2023-12-07T22:43:37Z)
• Equivariant Adaptation of Large Pretrained Models [20.687626756753563]
  We show that a canonicalization network can effectively be used to make a large pretrained network equivariant. Using dataset-dependent priors to inform the canonicalization function, we are able to make large pretrained models equivariant while maintaining their performance.
  arXiv  Detail & Related papers  (2023-10-02T21:21:28Z)
• Scalable and adaptive variational Bayes methods for Hawkes processes [4.580983642743026]
  We propose a novel sparsity-inducing procedure, and derive an adaptive mean-field variational algorithm for the popular sigmoid Hawkes processes. Our algorithm is parallelisable and therefore computationally efficient in high-dimensional setting.
  arXiv  Detail & Related papers  (2022-12-01T05:35:32Z)
• Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation [88.14365009076907]
  Iterative refinement is a useful paradigm for representation learning. We develop an implicit differentiation approach that improves the stability and tractability of training.
  arXiv  Detail & Related papers  (2022-07-02T10:00:35Z)
• Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation [110.09855163856326]
  This paper is about the problem of learning a policy for generating an object from a sequence of actions. We propose GFlowNet, based on a view of the generative process as a flow network. We prove that any global minimum of the proposed objectives yields a policy which samples from the desired distribution.
  arXiv  Detail & Related papers  (2021-06-08T14:21:10Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Regularizing Towards Permutation Invariance in Recurrent Models [26.36835670113303]
  We show that RNNs can be regularized towards permutation invariance, and that this can result in compact models. Existing solutions mostly suggest restricting the learning problem to hypothesis classes which are permutation invariant by design. We show that our method outperforms other permutation invariant approaches on synthetic and real world datasets.
  arXiv  Detail & Related papers  (2020-10-25T07:46:51Z)
• Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
  We show that common optimization methods lead to poor variational approximations if the problem is moderately large. Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
  arXiv  Detail & Related papers  (2020-09-01T19:12:11Z)
• MINA: Convex Mixed-Integer Programming for Non-Rigid Shape Alignment [77.38594866794429]
  convex mixed-integer programming formulation for non-rigid shape matching. We propose a novel shape deformation model based on an efficient low-dimensional discrete model.
  arXiv  Detail & Related papers  (2020-02-28T09:54:06Z)
• Learning with Differentiable Perturbed Optimizers [54.351317101356614]
  We propose a systematic method to transform operations into operations that are differentiable and never locally constant. Our approach relies on perturbeds, and can be used readily together with existing solvers. We show how this framework can be connected to a family of losses developed in structured prediction, and give theoretical guarantees for their use in learning tasks.
  arXiv  Detail & Related papers  (2020-02-20T11:11:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.