Meta-Learning Divergences of Variational Inference

• URL: http://arxiv.org/abs/2007.02912v2
• Date: Tue, 22 Jun 2021 20:28:21 GMT
• Title: Meta-Learning Divergences of Variational Inference
• Authors: Ruqi Zhang, Yingzhen Li, Christopher De Sa, Sam Devlin, Cheng Zhang
• Abstract summary: Variational inference (VI) plays an essential role in approximate Bayesian inference. We propose a meta-learning algorithm to learn the divergence metric suited for the task of interest. We demonstrate our approach outperforms standard VI on Gaussian mixture distribution approximation.
• Score: 49.164944557174294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Variational inference (VI) plays an essential role in approximate Bayesian inference due to its computational efficiency and broad applicability. Crucial to the performance of VI is the selection of the associated divergence measure, as VI approximates the intractable distribution by minimizing this divergence. In this paper we propose a meta-learning algorithm to learn the divergence metric suited for the task of interest, automating the design of VI methods. In addition, we learn the initialization of the variational parameters without additional cost when our method is deployed in the few-shot learning scenarios. We demonstrate our approach outperforms standard VI on Gaussian mixture distribution approximation, Bayesian neural network regression, image generation with variational autoencoders and recommender systems with a partial variational autoencoder.

Related papers

• Recursive Learning of Asymptotic Variational Objectives [49.69399307452126]
  General state-space models (SSMs) are widely used in statistical machine learning and are among the most classical generative models for sequential time-series data. Online sequential IWAE (OSIWAE) allows for online learning of both model parameters and a Markovian recognition model for inferring latent states. This approach is more theoretically well-founded than recently proposed online variational SMC methods.
  arXiv  Detail & Related papers  (2024-11-04T16:12:37Z)
• Nonparametric Automatic Differentiation Variational Inference with Spline Approximation [7.5620760132717795]
  We develop a nonparametric approximation approach that enables flexible posterior approximation for distributions with complicated structures. Compared with widely-used nonparametrical inference methods, the proposed method is easy to implement and adaptive to various data structures. Experiments demonstrate the efficiency of the proposed method in approximating complex posterior distributions and improving the performance of generative models with incomplete data.
  arXiv  Detail & Related papers  (2024-03-10T20:22:06Z)
• Semi-Implicit Variational Inference via Score Matching [9.654640791869431]
  Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families. Current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. We propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching.
  arXiv  Detail & Related papers  (2023-08-19T13:32:54Z)
• Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models [18.134776677795077]
  We focus on the interplay between variational inference (VI) and the learning target. We design a hybrid training procedure to bring the best of both worlds: it leverages conjugate-computation VI for inference. We empirically demonstrate the effectiveness of our proposal across a wide range of data sets.
  arXiv  Detail & Related papers  (2023-06-07T07:15:08Z)
• An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization [52.44068740462729]
  We present an information-theoretic perspective on the VICReg objective. We derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. We introduce a family of SSL methods derived from information-theoretic principles that outperform existing SSL techniques.
  arXiv  Detail & Related papers  (2023-03-01T16:36:25Z)
• Manifold Gaussian Variational Bayes on the Precision Matrix [70.44024861252554]
  We propose an optimization algorithm for Variational Inference (VI) in complex models. We develop an efficient algorithm for Gaussian Variational Inference whose updates satisfy the positive definite constraint on the variational covariance matrix. Due to its black-box nature, MGVBP stands as a ready-to-use solution for VI in complex models.
  arXiv  Detail & Related papers  (2022-10-26T10:12:31Z)
• GFlowNets and variational inference [64.22223306224903]
  This paper builds bridges between two families of probabilistic algorithms:hierarchical variational inference (VI) and generative flow networks (GFlowNets) We demonstrate that, in certain cases, VI algorithms are equivalent to special cases of GFlowNets in the sense of equality of expected gradients of their learning objectives.
  arXiv  Detail & Related papers  (2022-10-02T17:41:01Z)
• Amortized Variational Inference: A Systematic Review [0.0]
  The core principle of Variational Inference (VI) is to convert the statistical inference problem of computing complex posterior probability densities into a tractable optimization problem. The traditional VI algorithm is not scalable to large data sets and is unable to readily infer out-of-bounds data points. Recent developments in the field, like black box-, and amortized-VI, have helped address these issues.
  arXiv  Detail & Related papers  (2022-09-22T09:45:10Z)
• A Variational Bayesian Approach to Learning Latent Variables for Acoustic Knowledge Transfer [55.20627066525205]
  We propose a variational Bayesian (VB) approach to learning distributions of latent variables in deep neural network (DNN) models. Our proposed VB approach can obtain good improvements on target devices, and consistently outperforms 13 state-of-the-art knowledge transfer algorithms.
  arXiv  Detail & Related papers  (2021-10-16T15:54:01Z)
• An Introduction to Variational Inference [0.0]
  In this paper, we introduce the concept of Variational Inference (VI) VI is a popular method in machine learning that uses optimization techniques to estimate complex probability densities. We discuss the applications of VI to variational auto-encoders (VAE) and VAE-Generative Adversarial Network (VAE-GAN)
  arXiv  Detail & Related papers  (2021-08-30T09:40:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.