Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectives

• URL: http://arxiv.org/abs/2201.10989v1
• Date: Wed, 26 Jan 2022 15:04:03 GMT
• Title: Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectives
• Authors: Pierre-Alexandre Mattei and Jes Frellsen
• Abstract summary: We revisit the theory of importance weighted variational inference (IWVI) IWVI uses new variational bounds, known as Monte Carlo objectives (MCOs) We show that, in a precise sense, negative correlation reduces the variational gap.
• Score: 21.154936422150683
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We revisit the theory of importance weighted variational inference (IWVI), a promising strategy for learning latent variable models. IWVI uses new variational bounds, known as Monte Carlo objectives (MCOs), obtained by replacing intractable integrals by Monte Carlo estimates -- usually simply obtained via importance sampling. Burda, Grosse and Salakhutdinov (2016) showed that increasing the number of importance samples provably tightens the gap between the bound and the likelihood. Inspired by this simple monotonicity theorem, we present a series of nonasymptotic results that link properties of Monte Carlo estimates to tightness of MCOs. We challenge the rationale that smaller Monte Carlo variance leads to better bounds. We confirm theoretically the empirical findings of several recent papers by showing that, in a precise sense, negative correlation reduces the variational gap. We also generalise the original monotonicity theorem by considering non-uniform weights. We discuss several practical consequences of our theoretical results. Our work borrows many ideas and results from the theory of stochastic orders.

Related papers

• Multiply-Robust Causal Change Attribution [15.501106533308798]
  We develop a new estimation strategy that combines regression and re-weighting methods to quantify the contribution of each causal mechanism. Our method demonstrates excellent performance in Monte Carlo simulations, and we show its usefulness in an empirical application.
  arXiv  Detail & Related papers  (2024-04-12T22:57:01Z)
• Variational Inference for GARCH-family Models [84.84082555964086]
  Variational Inference is a robust approach for Bayesian inference in machine learning models. We show that Variational Inference is an attractive, remarkably well-calibrated, and competitive method for Bayesian learning.
  arXiv  Detail & Related papers  (2023-10-05T10:21:31Z)
• PAVI: Plate-Amortized Variational Inference [55.975832957404556]
  Inference is challenging for large population studies where millions of measurements are performed over a cohort of hundreds of subjects. This large cardinality renders off-the-shelf Variational Inference (VI) computationally impractical. In this work, we design structured VI families that efficiently tackle large population studies.
  arXiv  Detail & Related papers  (2023-08-30T13:22:20Z)
• Stochastic automatic differentiation for Monte Carlo processes [1.1279808969568252]
  We consider the extension of Automatic Differentiation (AD) techniques to Monte Carlo process. We show that the Hamiltonian approach can be understood as a change of variables of the reweighting approach.
  arXiv  Detail & Related papers  (2023-07-28T08:59:01Z)
• When can Regression-Adjusted Control Variates Help? Rare Events, Sobolev Embedding and Minimax Optimality [10.21792151799121]
  We show that a machine learning-based estimator can be used to mitigate the variance of Monte Carlo sampling. In the presence of rare and extreme events, a truncated version of the Monte Carlo algorithm can achieve the minimax optimal rate.
  arXiv  Detail & Related papers  (2023-05-25T23:09:55Z)
• Quasi-Monte Carlo Graph Random Features [38.762964164013226]
  We present a novel mechanism to improve the accuracy of graph random features (GRFs) Our method induces negative correlations between the lengths of the algorithm's random walks by imposing antithetic termination. We derive strong theoretical guarantees on the properties of these quasi-Monte Carlo GRFs.
  arXiv  Detail & Related papers  (2023-05-21T14:12:02Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Recursive Monte Carlo and Variational Inference with Auxiliary Variables [64.25762042361839]
  Recursive auxiliary-variable inference (RAVI) is a new framework for exploiting flexible proposals. RAVI generalizes and unifies several existing methods for inference with expressive expressive families. We show RAVI's design framework and theorems by using them to analyze and improve upon Salimans et al.'s Markov Chain Variational Inference.
  arXiv  Detail & Related papers  (2022-03-05T23:52:40Z)
• Deconfounded Score Method: Scoring DAGs with Dense Unobserved Confounding [101.35070661471124]
  We show that unobserved confounding leaves a characteristic footprint in the observed data distribution that allows for disentangling spurious and causal effects. We propose an adjusted score-based causal discovery algorithm that may be implemented with general-purpose solvers and scales to high-dimensional problems.
  arXiv  Detail & Related papers  (2021-03-28T11:07:59Z)
• Annealed Flow Transport Monte Carlo [91.20263039913912]
  Annealed Flow Transport (AFT) builds upon Annealed Importance Sampling (AIS) and Sequential Monte Carlo (SMC) AFT relies on NF which is learned sequentially to push particles towards the successive targets. We show that a continuous-time scaling limit of the population version of AFT is given by a Feynman--Kac measure.
  arXiv  Detail & Related papers  (2021-02-15T12:05:56Z)
• The variance of relative surprisal as single-shot quantifier [0.0]
  We show that (relative) surprisal gives sufficient conditions for approximate state-transitions between pairs of quantum states in single-shot setting. We further derive a simple and physically appealing axiomatic single-shot characterization of (relative) entropy.
  arXiv  Detail & Related papers  (2020-09-17T16:06:54Z)
• Scalable Control Variates for Monte Carlo Methods via Stochastic Optimization [62.47170258504037]
  This paper presents a framework that encompasses and generalizes existing approaches that use controls, kernels and neural networks. Novel theoretical results are presented to provide insight into the variance reduction that can be achieved, and an empirical assessment, including applications to Bayesian inference, is provided in support.
  arXiv  Detail & Related papers  (2020-06-12T22:03:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.