Variational Inference with Gaussian Score Matching

• URL: http://arxiv.org/abs/2307.07849v1
• Date: Sat, 15 Jul 2023 16:57:48 GMT
• Title: Variational Inference with Gaussian Score Matching
• Authors: Chirag Modi, Charles Margossian, Yuling Yao, Robert Gower, David Blei and Lawrence Saul
• Abstract summary: We present a new approach to VI based on the principle of score matching. We develop score matching VI, an iterative algorithm that seeks to match the scores between the variational approximation and the exact posterior. In all of our studies we find that GSM-VI is faster than BBVI, but without sacrificing accuracy.
• Score: 1.2233362977312945
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Variational inference (VI) is a method to approximate the computationally intractable posterior distributions that arise in Bayesian statistics. Typically, VI fits a simple parametric distribution to the target posterior by minimizing an appropriate objective such as the evidence lower bound (ELBO). In this work, we present a new approach to VI based on the principle of score matching, that if two distributions are equal then their score functions (i.e., gradients of the log density) are equal at every point on their support. With this, we develop score matching VI, an iterative algorithm that seeks to match the scores between the variational approximation and the exact posterior. At each iteration, score matching VI solves an inner optimization, one that minimally adjusts the current variational estimate to match the scores at a newly sampled value of the latent variables. We show that when the variational family is a Gaussian, this inner optimization enjoys a closed form solution, which we call Gaussian score matching VI (GSM-VI). GSM-VI is also a ``black box'' variational algorithm in that it only requires a differentiable joint distribution, and as such it can be applied to a wide class of models. We compare GSM-VI to black box variational inference (BBVI), which has similar requirements but instead optimizes the ELBO. We study how GSM-VI behaves as a function of the problem dimensionality, the condition number of the target covariance matrix (when the target is Gaussian), and the degree of mismatch between the approximating and exact posterior distribution. We also study GSM-VI on a collection of real-world Bayesian inference problems from the posteriorDB database of datasets and models. In all of our studies we find that GSM-VI is faster than BBVI, but without sacrificing accuracy. It requires 10-100x fewer gradient evaluations to obtain a comparable quality of approximation.

Related papers

• EigenVI: score-based variational inference with orthogonal function expansions [23.696028065251497]
  EigenVI is an eigenvalue-based approach for black-box variational inference (BBVI) We use EigenVI to approximate a variety of target distributions, including a benchmark suite of Bayesian models from posteriordb.
  arXiv  Detail & Related papers  (2024-10-31T15:48:34Z)
• Batch, match, and patch: low-rank approximations for score-based variational inference [8.840147522046651]
  Black-box variational inference scales poorly to high dimensional problems. We extend the batch-and-match framework for score-based BBVI. We evaluate this approach on a variety of synthetic target distributions and real-world problems in high-dimensional inference.
  arXiv  Detail & Related papers  (2024-10-29T17:42:56Z)
• You Only Accept Samples Once: Fast, Self-Correcting Stochastic Variational Inference [0.0]
  YOASOVI is an algorithm for performing fast, self-correcting intuition optimization for Variational Inference (VI) on large Bayesian heirarchical models. To accomplish this, we take advantage of available information on the objective function used for VI at each iteration and replace regular Monte Carlo sampling with acceptance sampling.
  arXiv  Detail & Related papers  (2024-06-05T01:28:53Z)
• Batch and match: black-box variational inference with a score-based divergence [26.873037094654826]
  We propose batch and match (BaM) as an alternative approach to blackbox variational inference (BBVI) based on a score-based divergence. We show that BaM converges in fewer evaluations than leading implementations of BBVI based on ELBO.
  arXiv  Detail & Related papers  (2024-02-22T18:20:22Z)
• Distributed Extra-gradient with Optimal Complexity and Communication Guarantees [60.571030754252824]
  We consider monotone variational inequality (VI) problems in multi-GPU settings where multiple processors/workers/clients have access to local dual vectors. Extra-gradient, which is a de facto algorithm for monotone VI problems, has not been designed to be communication-efficient. We propose a quantized generalized extra-gradient (Q-GenX), which is an unbiased and adaptive compression method tailored to solve VIs.
  arXiv  Detail & Related papers  (2023-08-17T21:15:04Z)
• Manifold Gaussian Variational Bayes on the Precision Matrix [70.44024861252554]
  We propose an optimization algorithm for Variational Inference (VI) in complex models. We develop an efficient algorithm for Gaussian Variational Inference whose updates satisfy the positive definite constraint on the variational covariance matrix. Due to its black-box nature, MGVBP stands as a ready-to-use solution for VI in complex models.
  arXiv  Detail & Related papers  (2022-10-26T10:12:31Z)
• Efficient Approximate Kernel Based Spike Sequence Classification [56.2938724367661]
  Machine learning models, such as SVM, require a definition of distance/similarity between pairs of sequences. Exact methods yield better classification performance, but they pose high computational costs. We propose a series of ways to improve the performance of the approximate kernel in order to enhance its predictive performance.
  arXiv  Detail & Related papers  (2022-09-11T22:44:19Z)
• Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling? [59.820507600960745]
  We propose a new GCP meta-layer that uses SVD in the forward pass, and Pad'e Approximants in the backward propagation to compute the gradients. The proposed meta-layer has been integrated into different CNN models and achieves state-of-the-art performances on both large-scale and fine-grained datasets.
  arXiv  Detail & Related papers  (2021-05-06T08:03:45Z)
• Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds [72.63031036770425]
  We propose differentially private (DP) algorithms for bound non-dimensional optimization. We demonstrate two popular deep learning methods on the empirical advantages over standard gradient methods.
  arXiv  Detail & Related papers  (2020-06-24T06:01:24Z)
• Non-Adaptive Adaptive Sampling on Turnstile Streams [57.619901304728366]
  We give the first relative-error algorithms for column subset selection, subspace approximation, projective clustering, and volume on turnstile streams that use space sublinear in $n$. Our adaptive sampling procedure has a number of applications to various data summarization problems that either improve state-of-the-art or have only been previously studied in the more relaxed row-arrival model.
  arXiv  Detail & Related papers  (2020-04-23T05:00:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.