Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization

• URL: http://arxiv.org/abs/2012.05942v2
• Date: Tue, 23 Feb 2021 20:15:35 GMT
• Title: Convex Potential Flows: Universal Probability Distributions with Optimal Transport and Convex Optimization
• Authors: Chin-Wei Huang, Ricky T. Q. Chen, Christos Tsirigotis, Aaron Courville
• Abstract summary: This paper introduces Convex Potential Flows (CP-Flow), a natural and efficient parameterization of invertible models. CP-Flows are the gradient map of a strongly convex neural potential function. We show that CP-Flow performs competitively on standard benchmarks of density estimation and variational inference.
• Score: 8.683116789109462
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flow-based models are powerful tools for designing probabilistic models with tractable density. This paper introduces Convex Potential Flows (CP-Flow), a natural and efficient parameterization of invertible models inspired by the optimal transport (OT) theory. CP-Flows are the gradient map of a strongly convex neural potential function. The convexity implies invertibility and allows us to resort to convex optimization to solve the convex conjugate for efficient inversion. To enable maximum likelihood training, we derive a new gradient estimator of the log-determinant of the Jacobian, which involves solving an inverse-Hessian vector product using the conjugate gradient method. The gradient estimator has constant-memory cost, and can be made effectively unbiased by reducing the error tolerance level of the convex optimization routine. Theoretically, we prove that CP-Flows are universal density approximators and are optimal in the OT sense. Our empirical results show that CP-Flow performs competitively on standard benchmarks of density estimation and variational inference.

Related papers

• Semi-Implicit Functional Gradient Flow [30.32233517392456]
  We propose a functional gradient ParVI method that uses perturbed particles as the approximation family. The corresponding functional gradient flow, which can be estimated via denoising score matching, exhibits strong theoretical convergence guarantee.
  arXiv  Detail & Related papers  (2024-10-23T15:00:30Z)
• Flow-based Distributionally Robust Optimization [23.232731771848883]
  We present a framework, called $textttFlowDRO$, for solving flow-based distributionally robust optimization (DRO) problems with Wasserstein uncertainty sets. We aim to find continuous worst-case distribution (also called the Least Favorable Distribution, LFD) and sample from it. We demonstrate its usage in adversarial learning, distributionally robust hypothesis testing, and a new mechanism for data-driven distribution perturbation differential privacy.
  arXiv  Detail & Related papers  (2023-10-30T03:53:31Z)
• $\mathcal{C}^k$-continuous Spline Approximation with TensorFlow Gradient Descent Optimizers [2.0305676256390934]
  We present an "out-of-the-box" application of Machine Learning (ML)s for an industrial optimization problem. We introduce a piecewise model (spline) for fitting of $mathcalCk$-continuos functions, which can be deployed in a cam approximation setting. We then use the gradient descent optimization provided by the machine learning framework to optimize the model parameters with respect to approximation quality and $mathcalCk$-continuity.
  arXiv  Detail & Related papers  (2023-03-22T10:52:21Z)
• Universality of parametric Coupling Flows over parametric diffeomorphisms [13.39432329310336]
  Invertible neural networks based on Coupling Flows CFlows) have various applications such as image synthesis and data compression. We prove that CFlows can approximate any diffeomorphism in Ck-norm if its layers can approximate certain single-coordinate transforms.
  arXiv  Detail & Related papers  (2022-02-07T02:10:32Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• Zeroth-Order Hybrid Gradient Descent: Towards A Principled Black-Box Optimization Framework [100.36569795440889]
  This work is on the iteration of zero-th-order (ZO) optimization which does not require first-order information. We show that with a graceful design in coordinate importance sampling, the proposed ZO optimization method is efficient both in terms of complexity as well as as function query cost.
  arXiv  Detail & Related papers  (2020-12-21T17:29:58Z)
• Self Normalizing Flows [65.73510214694987]
  We propose a flexible framework for training normalizing flows by replacing expensive terms in the gradient by learned approximate inverses at each layer. This reduces the computational complexity of each layer's exact update from $mathcalO(D3)$ to $mathcalO(D2)$. We show experimentally that such models are remarkably stable and optimize to similar data likelihood values as their exact gradient counterparts.
  arXiv  Detail & Related papers  (2020-11-14T09:51:51Z)
• OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport [8.468007443062751]
  A normalizing flow is an invertible mapping between an arbitrary probability distribution and a standard normal distribution. OT-Flow tackles two critical computational challenges that limit a more widespread use of CNFs. On five high-dimensional density estimation and generative modeling tasks, OT-Flow performs competitively to state-of-the-art CNFs.
  arXiv  Detail & Related papers  (2020-05-29T22:31:10Z)
• Gaussianization Flows [113.79542218282282]
  We propose a new type of normalizing flow model that enables both efficient iteration of likelihoods and efficient inversion for sample generation. Because of this guaranteed expressivity, they can capture multimodal target distributions without compromising the efficiency of sample generation.
  arXiv  Detail & Related papers  (2020-03-04T08:15:06Z)
• Learning Likelihoods with Conditional Normalizing Flows [54.60456010771409]
  Conditional normalizing flows (CNFs) are efficient in sampling and inference. We present a study of CNFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p(y|x)
  arXiv  Detail & Related papers  (2019-11-29T19:17:58Z)
• A Near-Optimal Gradient Flow for Learning Neural Energy-Based Models [93.24030378630175]
  We propose a novel numerical scheme to optimize the gradient flows for learning energy-based models (EBMs) We derive a second-order Wasserstein gradient flow of the global relative entropy from Fokker-Planck equation. Compared with existing schemes, Wasserstein gradient flow is a smoother and near-optimal numerical scheme to approximate real data densities.
  arXiv  Detail & Related papers  (2019-10-31T02:26:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.