Semi-Implicit Variational Inference via Score Matching

• URL: http://arxiv.org/abs/2308.10014v1
• Date: Sat, 19 Aug 2023 13:32:54 GMT
• Title: Semi-Implicit Variational Inference via Score Matching
• Authors: Longlin Yu, Cheng Zhang
• Abstract summary: Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families. Current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. We propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching.
• Score: 9.654640791869431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families by considering implicit variational distributions defined in a hierarchical manner. However, due to the intractable densities of variational distributions, current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. In this paper, we propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching. Leveraging the hierarchical structure of semi-implicit variational families, the score matching objective allows a minimax formulation where the intractable variational densities can be naturally handled with denoising score matching. We show that SIVI-SM closely matches the accuracy of MCMC and outperforms ELBO-based SIVI methods in a variety of Bayesian inference tasks.

Related papers

• Particle Semi-Implicit Variational Inference [2.555222031881788]
  Semi-implicit variational inference (SIVI) enriches the expressiveness of variational families by utilizing a kernel and a mixing distribution. Existing SIVI methods parameterize the mixing distribution using implicit distributions, leading to intractable variational densities. We propose a novel method for SIVI called Particle Variational Inference (PVI) which employs empirical measures to approximate the optimal mixing distributions.
  arXiv  Detail & Related papers  (2024-06-30T10:21:41Z)
• Kernel Semi-Implicit Variational Inference [27.61976547543748]
  Semi-implicit variational inference (SIVI) extends traditional variational families with semi-implicit distributions defined in a hierarchical manner. A recent advancement in SIVI, named SIVI-SM, utilizes an alternative score matching objective made tractable via a minimax formulation. We propose kernel SIVI (KSIVI), a variant of SIVI-SM that eliminates the need for lower-level optimization through kernel tricks.
  arXiv  Detail & Related papers  (2024-05-29T11:21:25Z)
• Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration [20.820242501141834]
  Semi-implicit variational inference (SIVI) has been introduced to expand the analytical variational families. We propose hierarchical semi-implicit variational inference, called HSIVI, which generalizes SIVI to allow more expressive multi-layer construction.
  arXiv  Detail & Related papers  (2023-10-26T04:52:28Z)
• Conformal Inference for Invariant Risk Minimization [12.049545417799125]
  The application of machine learning models can be significantly impeded by the occurrence of distributional shifts. One way to tackle this problem is to use invariant learning, such as invariant risk minimization (IRM), to acquire an invariant representation. This paper develops methods for obtaining distribution-free prediction regions to describe uncertainty estimates for invariant representations.
  arXiv  Detail & Related papers  (2023-05-22T03:48:38Z)
• An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization [52.44068740462729]
  We present an information-theoretic perspective on the VICReg objective. We derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. We introduce a family of SSL methods derived from information-theoretic principles that outperform existing SSL techniques.
  arXiv  Detail & Related papers  (2023-03-01T16:36:25Z)
• Meta-Causal Feature Learning for Out-of-Distribution Generalization [71.38239243414091]
  This paper presents a balanced meta-causal learner (BMCL), which includes a balanced task generation module (BTG) and a meta-causal feature learning module (MCFL) BMCL effectively identifies the class-invariant visual regions for classification and may serve as a general framework to improve the performance of the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-08-22T09:07:02Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• Semi-Supervised Learning with Variational Bayesian Inference and Maximum Uncertainty Regularization [62.21716612888669]
  We propose two generic methods for improving semi-supervised learning (SSL) The first integrates weight perturbation (WP) into existing "consistency regularization" (CR) based methods. The second method proposes a novel consistency loss called "maximum uncertainty regularization" (MUR)
  arXiv  Detail & Related papers  (2020-12-03T09:49:35Z)
• Meta-Learning Divergences of Variational Inference [49.164944557174294]
  Variational inference (VI) plays an essential role in approximate Bayesian inference. We propose a meta-learning algorithm to learn the divergence metric suited for the task of interest. We demonstrate our approach outperforms standard VI on Gaussian mixture distribution approximation.
  arXiv  Detail & Related papers  (2020-07-06T17:43:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.