Variational Inference with Holder Bounds

• URL: http://arxiv.org/abs/2111.02947v1
• Date: Thu, 4 Nov 2021 15:35:47 GMT
• Title: Variational Inference with Holder Bounds
• Authors: Junya Chen, Danni Lu, Zidi Xiu, Ke Bai, Lawrence Carin, Chenyang Tao
• Abstract summary: We present a careful analysis of the thermodynamic variational objective (TVO) We reveal how the pathological geometry of thermodynamic curves negatively affects TVO. This motivates our new VI objectives, named the Holder bounds, which flatten the thermodynamic curves and promise to achieve a one-step approximation of the exact marginal log-likelihood.
• Score: 68.8008396694788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The recent introduction of thermodynamic integration techniques has provided a new framework for understanding and improving variational inference (VI). In this work, we present a careful analysis of the thermodynamic variational objective (TVO), bridging the gap between existing variational objectives and shedding new insights to advance the field. In particular, we elucidate how the TVO naturally connects the three key variational schemes, namely the importance-weighted VI, Renyi-VI, and MCMC-VI, which subsumes most VI objectives employed in practice. To explain the performance gap between theory and practice, we reveal how the pathological geometry of thermodynamic curves negatively affects TVO. By generalizing the integration path from the geometric mean to the weighted Holder mean, we extend the theory of TVO and identify new opportunities for improving VI. This motivates our new VI objectives, named the Holder bounds, which flatten the thermodynamic curves and promise to achieve a one-step approximation of the exact marginal log-likelihood. A comprehensive discussion on the choices of numerical estimators is provided. We present strong empirical evidence on both synthetic and real-world datasets to support our claims.

Related papers

• Learning with Importance Weighted Variational Inference: Asymptotics for Gradient Estimators of the VR-IWAE Bound [3.115375810642661]
  Several popular variational bounds involving importance weighting ideas have been proposed to generalize and improve on the Evidence Lower BOund. The VR-IWAE bound was introduced as a variational bound that unifies the ELBO, IWAE and VR bounds methodologies.
  arXiv  Detail & Related papers  (2024-10-15T20:09:06Z)
• On the Joint Interaction of Models, Data, and Features [82.60073661644435]
  We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
  arXiv  Detail & Related papers  (2023-06-07T21:35:26Z)
• On the Convergence of Coordinate Ascent Variational Inference [11.166959724276337]
  We consider the common coordinate ascent variational inference (CAVI) algorithm for implementing the mean-field (MF) VI. We provide general conditions for certifying global or local exponential convergence of CAVI. New notion of generalized correlation for characterizing the interaction between the constituting blocks in influencing the VI objective functional is introduced.
  arXiv  Detail & Related papers  (2023-06-01T20:19:30Z)
• Variational Distillation for Multi-View Learning [104.17551354374821]
  We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning. Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
  arXiv  Detail & Related papers  (2022-06-20T03:09:46Z)
• Handling Distribution Shifts on Graphs: An Invariance Perspective [78.31180235269035]
  We formulate the OOD problem on graphs and develop a new invariant learning approach, Explore-to-Extrapolate Risk Minimization (EERM) EERM resorts to multiple context explorers that are adversarially trained to maximize the variance of risks from multiple virtual environments. We prove the validity of our method by theoretically showing its guarantee of a valid OOD solution.
  arXiv  Detail & Related papers  (2022-02-05T02:31:01Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Discovering Latent Causal Variables via Mechanism Sparsity: A New Principle for Nonlinear ICA [81.4991350761909]
  Independent component analysis (ICA) refers to an ensemble of methods which formalize this goal and provide estimation procedure for practical application. We show that the latent variables can be recovered up to a permutation if one regularizes the latent mechanisms to be sparse.
  arXiv  Detail & Related papers  (2021-07-21T14:22:14Z)
• Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification [41.02729491273057]
  The Information Bottleneck (IB) provides an information theoretic principle for representation learning. We present a new strategy, Variational Self-Distillation (VSD), which provides a scalable, flexible and analytic solution. We also introduce two other strategies, Variational Cross-Distillation (VCD) and Variational Mutual-Learning (VML)
  arXiv  Detail & Related papers  (2021-04-07T02:19:41Z)
• All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference [42.05882835476882]
  We propose an exponential family interpretation of the geometric mixture curve underlying the Thermodynamic Variational Objective (TVO) We propose to choose intermediate distributions using equal spacing in the moment parameters of our exponential family, which matches grid search performance and allows the schedule to adaptively update over the course of training.
  arXiv  Detail & Related papers  (2020-07-01T17:46:49Z)
• Particle-based Energetic Variational Inference [4.079427359693159]
  We introduce a new variational inference (VI) framework, called energetic variational inference (EVI) We derive many existing Particle-based Variational Inference (ParVI) methods, including the popular Stein Variational Gradient Descent (SVGD) approach. We propose a new particle-based EVI scheme, which performs the particle-based approximation of the density first and then uses the approximated density in the variational procedure.
  arXiv  Detail & Related papers  (2020-04-14T12:14:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.