Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians

• URL: http://arxiv.org/abs/2406.04012v2
• Date: Mon, 10 Jun 2024 09:32:49 GMT
• Title: Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians
• Authors: Tom Huix, Anna Korba, Alain Durmus, Eric Moulines,
• Abstract summary: Variational inference (VI) is a popular approach in Bayesian inference. This work aims to contribute to the theoretical study of VI in the non-Gaussian case.
• Score: 27.20127082606962
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Variational inference (VI) is a popular approach in Bayesian inference, that looks for the best approximation of the posterior distribution within a parametric family, minimizing a loss that is typically the (reverse) Kullback-Leibler (KL) divergence. Despite its empirical success, the theoretical properties of VI have only received attention recently, and mostly when the parametric family is the one of Gaussians. This work aims to contribute to the theoretical study of VI in the non-Gaussian case by investigating the setting of Mixture of Gaussians with fixed covariance and constant weights. In this view, VI over this specific family can be casted as the minimization of a Mollified relative entropy, i.e. the KL between the convolution (with respect to a Gaussian kernel) of an atomic measure supported on Diracs, and the target distribution. The support of the atomic measure corresponds to the localization of the Gaussian components. Hence, solving variational inference becomes equivalent to optimizing the positions of the Diracs (the particles), which can be done through gradient descent and takes the form of an interacting particle system. We study two sources of error of variational inference in this context when optimizing the mollified relative entropy. The first one is an optimization result, that is a descent lemma establishing that the algorithm decreases the objective at each iteration. The second one is an approximation error, that upper bounds the objective between an optimal finite mixture and the target distribution.

Related papers

• Straightness of Rectified Flow: A Theoretical Insight into Wasserstein Convergence [54.580605276017096]
  Diffusion models have emerged as a powerful tool for image generation and denoising. Recently, Liu et al. designed a novel alternative generative model Rectified Flow (RF) RF aims to learn straight flow trajectories from noise to data using a sequence of convex optimization problems.
  arXiv  Detail & Related papers  (2024-10-19T02:36:11Z)
• Variance-Reducing Couplings for Random Features [57.73648780299374]
  Random features (RFs) are a popular technique to scale up kernel methods in machine learning. We find couplings to improve RFs defined on both Euclidean and discrete input spaces. We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm.
  arXiv  Detail & Related papers  (2024-05-26T12:25:09Z)
• Batch and match: black-box variational inference with a score-based divergence [26.873037094654826]
  We propose batch and match (BaM) as an alternative approach to blackbox variational inference (BBVI) based on a score-based divergence. We show that BaM converges in fewer evaluations than leading implementations of BBVI based on ELBO.
  arXiv  Detail & Related papers  (2024-02-22T18:20:22Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)
• Forward-backward Gaussian variational inference via JKO in the Bures-Wasserstein Space [19.19325201882727]
  Variational inference (VI) seeks to approximate a target distribution $pi$ by an element of a tractable family of distributions. We develop the Forward-Backward Gaussian Variational Inference (FB-GVI) algorithm to solve Gaussian VI. For our proposed algorithm, we obtain state-of-the-art convergence guarantees when $pi$ is log-smooth and log-concave.
  arXiv  Detail & Related papers  (2023-04-10T19:49:50Z)
• Riemannian optimization for non-centered mixture of scaled Gaussian distributions [17.855338784378]
  This paper studies the statistical model of the non-centered mixture of scaled Gaussian distributions (NC-MSG) Using the Fisher-Rao information geometry associated to this distribution, we derive a Riemannian gradient descent algorithm. A Nearest centroid classifier is implemented leveraging the KL divergence and its associated center of mass.
  arXiv  Detail & Related papers  (2022-09-07T17:22:20Z)
• How Good are Low-Rank Approximations in Gaussian Process Regression? [28.392890577684657]
  We provide guarantees for approximate Gaussian Process (GP) regression resulting from two common low-rank kernel approximations. We provide experiments on both simulated data and standard benchmarks to evaluate the effectiveness of our theoretical bounds.
  arXiv  Detail & Related papers  (2021-12-13T04:04:08Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• Spectral clustering under degree heterogeneity: a case for the random walk Laplacian [83.79286663107845]
  This paper shows that graph spectral embedding using the random walk Laplacian produces vector representations which are completely corrected for node degree. In the special case of a degree-corrected block model, the embedding concentrates about K distinct points, representing communities.
  arXiv  Detail & Related papers  (2021-05-03T16:36:27Z)
• Variational Transport: A Convergent Particle-BasedAlgorithm for Distributional Optimization [106.70006655990176]
  A distributional optimization problem arises widely in machine learning and statistics. We propose a novel particle-based algorithm, dubbed as variational transport, which approximately performs Wasserstein gradient descent. We prove that when the objective function satisfies a functional version of the Polyak-Lojasiewicz (PL) (Polyak, 1963) and smoothness conditions, variational transport converges linearly.
  arXiv  Detail & Related papers  (2020-12-21T18:33:13Z)
• How Good are Low-Rank Approximations in Gaussian Process Regression? [24.09582049403961]
  We provide guarantees for approximate Gaussian Process (GP) regression resulting from two common low-rank kernel approximations. We provide experiments on both simulated data and standard benchmarks to evaluate the effectiveness of our theoretical bounds.
  arXiv  Detail & Related papers  (2020-04-03T14:15:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.