SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows

• URL: http://arxiv.org/abs/2007.02731v2
• Date: Fri, 30 Oct 2020 17:08:53 GMT
• Title: SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
• Authors: Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, Max Welling
• Abstract summary: SurVAE Flows is a modular framework for composable transformations that encompasses VAEs and normalizing flows. We show that several recently proposed methods, including dequantization and augmented normalizing flows, can be expressed as SurVAE Flows.
• Score: 78.77808270452974
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Normalizing flows and variational autoencoders are powerful generative models that can represent complicated density functions. However, they both impose constraints on the models: Normalizing flows use bijective transformations to model densities whereas VAEs learn stochastic transformations that are non-invertible and thus typically do not provide tractable estimates of the marginal likelihood. In this paper, we introduce SurVAE Flows: A modular framework of composable transformations that encompasses VAEs and normalizing flows. SurVAE Flows bridge the gap between normalizing flows and VAEs with surjective transformations, wherein the transformations are deterministic in one direction -- thereby allowing exact likelihood computation, and stochastic in the reverse direction -- hence providing a lower bound on the corresponding likelihood. We show that several recently proposed methods, including dequantization and augmented normalizing flows, can be expressed as SurVAE Flows. Finally, we introduce common operations such as the max value, the absolute value, sorting and stochastic permutation as composable layers in SurVAE Flows.

Related papers

• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• Efficient CDF Approximations for Normalizing Flows [64.60846767084877]
  We build upon the diffeomorphic properties of normalizing flows to estimate the cumulative distribution function (CDF) over a closed region. Our experiments on popular flow architectures and UCI datasets show a marked improvement in sample efficiency as compared to traditional estimators.
  arXiv  Detail & Related papers  (2022-02-23T06:11:49Z)
• Sinusoidal Flow: A Fast Invertible Autoregressive Flow [0.0]
  We propose a new type of normalising flows that inherits the expressive power and triangular Jacobian from fully autoregressive flows. Experiments show that our Sinusoidal Flow is not only able to model complex distributions, but can also be reliably inverted to generate realistic-looking samples.
  arXiv  Detail & Related papers  (2021-10-26T01:20:29Z)
• Equivariant Discrete Normalizing Flows [10.867162810786361]
  We focus on building equivariant normalizing flows using discrete layers. We introduce two new equivariant flows: $G$-coupling Flows and $G$-Residual Flows. Our construction of $G$-Residual Flows are also universal, in the sense that we prove an $G$-equivariant diffeomorphism can be exactly mapped by a $G$-residual flow.
  arXiv  Detail & Related papers  (2021-10-16T20:16:00Z)
• Stochastic Normalizing Flows for Inverse Problems: a Markov Chains Viewpoint [0.45119235878273]
  We consider normalizing flows from a Markov chain point of view. We replace transition densities by general Markov kernels and establish proofs via Radon-Nikodym derivatives. The performance of the proposed conditional normalizing flow is demonstrated by numerical examples.
  arXiv  Detail & Related papers  (2021-09-23T13:44:36Z)
• Generalization of the Change of Variables Formula with Applications to Residual Flows [7.57024681220677]
  Normalizing flows leverage the Change of Variables Formula to define flexible density models. We introduce $mathcalL$-diffeomorphisms as generalized transformations which may violate these requirements on zero Lebesgue-measure sets. This relaxation allows the use of non-smooth activation functions such as ReLU.
  arXiv  Detail & Related papers  (2021-07-09T10:31:32Z)
• Generative Flows with Invertible Attentions [135.23766216657745]
  We introduce two types of invertible attention mechanisms for generative flow models. We exploit split-based attention mechanisms to learn the attention weights and input representations on every two splits of flow feature maps. Our method provides invertible attention modules with tractable Jacobian determinants, enabling seamless integration of it at any positions of the flow-based models.
  arXiv  Detail & Related papers  (2021-06-07T20:43:04Z)
• The Convolution Exponential and Generalized Sylvester Flows [82.18442368078804]
  This paper introduces a new method to build linear flows, by taking the exponential of a linear transformation. An important insight is that the exponential can be computed implicitly, which allows the use of convolutional layers. We show that the convolution exponential outperforms other linear transformations in generative flows on CIFAR10.
  arXiv  Detail & Related papers  (2020-06-02T19:43:36Z)
• Gaussianization Flows [113.79542218282282]
  We propose a new type of normalizing flow model that enables both efficient iteration of likelihoods and efficient inversion for sample generation. Because of this guaranteed expressivity, they can capture multimodal target distributions without compromising the efficiency of sample generation.
  arXiv  Detail & Related papers  (2020-03-04T08:15:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.