Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows

• URL: http://arxiv.org/abs/2107.02951v1
• Date: Wed, 7 Jul 2021 00:13:50 GMT
• Title: Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows
• Authors: Holden Lee, Chirag Pabbaraju, Anish Sevekari, Andrej Risteski
• Abstract summary: We show that any log-concave distribution can be approximated using well-conditioned affine-coupling flows. Our results also inform the practice of training affine couplings.
• Score: 20.022920482589324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Normalizing flows are a widely used class of latent-variable generative models with a tractable likelihood. Affine-coupling (Dinh et al, 2014-16) models are a particularly common type of normalizing flows, for which the Jacobian of the latent-to-observable-variable transformation is triangular, allowing the likelihood to be computed in linear time. Despite the widespread usage of affine couplings, the special structure of the architecture makes understanding their representational power challenging. The question of universal approximation was only recently resolved by three parallel papers (Huang et al.,2020;Zhang et al.,2020;Koehler et al.,2020) -- who showed reasonably regular distributions can be approximated arbitrarily well using affine couplings -- albeit with networks with a nearly-singular Jacobian. As ill-conditioned Jacobians are an obstacle for likelihood-based training, the fundamental question remains: which distributions can be approximated using well-conditioned affine coupling flows? In this paper, we show that any log-concave distribution can be approximated using well-conditioned affine-coupling flows. In terms of proof techniques, we uncover and leverage deep connections between affine coupling architectures, underdamped Langevin dynamics (a stochastic differential equation often used to sample from Gibbs measures) and H\'enon maps (a structured dynamical system that appears in the study of symplectic diffeomorphisms). Our results also inform the practice of training affine couplings: we approximate a padded version of the input distribution with iid Gaussians -- a strategy which Koehler et al.(2020) empirically observed to result in better-conditioned flows, but had hitherto no theoretical grounding. Our proof can thus be seen as providing theoretical evidence for the benefits of Gaussian padding when training normalizing flows.

Related papers

• Rectified Diffusion Guidance for Conditional Generation [62.00207951161297]
  We revisit the theory behind CFG and rigorously confirm that the improper configuration of the combination coefficients (i.e., the widely used summing-to-one version) brings about expectation shift of the generative distribution. We propose ReCFG with a relaxation on the guidance coefficients such that denoising with ReCFG strictly aligns with the diffusion theory. That way the rectified coefficients can be readily pre-computed via traversing the observed data, leaving the sampling speed barely affected.
  arXiv  Detail & Related papers  (2024-10-24T13:41:32Z)
• Straightness of Rectified Flow: A Theoretical Insight into Wasserstein Convergence [54.580605276017096]
  Diffusion models have emerged as a powerful tool for image generation and denoising. Recently, Liu et al. designed a novel alternative generative model Rectified Flow (RF) RF aims to learn straight flow trajectories from noise to data using a sequence of convex optimization problems.
  arXiv  Detail & Related papers  (2024-10-19T02:36:11Z)
• Applications of flow models to the generation of correlated lattice QCD ensembles [69.18453821764075]
  Machine-learned normalizing flows can be used in the context of lattice quantum field theory to generate statistically correlated ensembles of lattice gauge fields at different action parameters. This work demonstrates how these correlations can be exploited for variance reduction in the computation of observables.
  arXiv  Detail & Related papers  (2024-01-19T18:33:52Z)
• Generalized Schrödinger Bridge Matching [54.171931505066]
  Generalized Schr"odinger Bridge (GSB) problem setup is prevalent in many scientific areas both within and without machine learning. We propose Generalized Schr"odinger Bridge Matching (GSBM), a new matching algorithm inspired by recent advances. We show that such a generalization can be cast as solving conditional optimal control, for which variational approximations can be used.
  arXiv  Detail & Related papers  (2023-10-03T17:42:11Z)
• Normalizing flow sampling with Langevin dynamics in the latent space [12.91637880428221]
  Normalizing flows (NF) use a continuous generator to map a simple latent (e.g. Gaussian) distribution, towards an empirical target distribution associated with a training data set. Since standard NF implement differentiable maps, they may suffer from pathological behaviors when targeting complex distributions. This paper proposes a new Markov chain Monte Carlo algorithm to sample from the target distribution in the latent domain before transporting it back to the target domain.
  arXiv  Detail & Related papers  (2023-05-20T09:31:35Z)
• Whitening Convergence Rate of Coupling-based Normalizing Flows [1.1279808969568252]
  Existing work shows that such flows weakly converge to arbitrary data distributions. We prove that all coupling-based normalizing flows perform whitening of the data distribution. We derive corresponding convergence bounds that show a linear convergence rate in the depth of the flow.
  arXiv  Detail & Related papers  (2022-10-25T14:10:34Z)
• Equivariant Discrete Normalizing Flows [10.867162810786361]
  We focus on building equivariant normalizing flows using discrete layers. We introduce two new equivariant flows: $G$-coupling Flows and $G$-Residual Flows. Our construction of $G$-Residual Flows are also universal, in the sense that we prove an $G$-equivariant diffeomorphism can be exactly mapped by a $G$-residual flow.
  arXiv  Detail & Related papers  (2021-10-16T20:16:00Z)
• Deep Archimedean Copulas [98.96141706464425]
  ACNet is a novel differentiable neural network architecture that enforces structural properties. We show that ACNet is able to both approximate common Archimedean Copulas and generate new copulas which may provide better fits to data.
  arXiv  Detail & Related papers  (2020-12-05T22:58:37Z)
• Representational aspects of depth and conditioning in normalizing flows [33.4333537858003]
  We show that representationally the choice of partition is not a bottleneck for depth. We also show that shallow affine coupling networks are universal approximators in Wasserstein distance.
  arXiv  Detail & Related papers  (2020-10-02T18:15:45Z)
• Gaussianization Flows [113.79542218282282]
  We propose a new type of normalizing flow model that enables both efficient iteration of likelihoods and efficient inversion for sample generation. Because of this guaranteed expressivity, they can capture multimodal target distributions without compromising the efficiency of sample generation.
  arXiv  Detail & Related papers  (2020-03-04T08:15:06Z)
• Latent Variable Modelling with Hyperbolic Normalizing Flows [35.1659722563025]
  We introduce a novel normalizing flow over hyperbolic VAEs and Euclidean normalizing flows. Our approach achieves improved performance on density estimation, as well as reconstruction of real-world graph data.
  arXiv  Detail & Related papers  (2020-02-15T07:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.