Variational Inference with Mixtures of Isotropic Gaussians

• URL: http://arxiv.org/abs/2506.13613v1
• Date: Mon, 16 Jun 2025 15:42:15 GMT
• Title: Variational Inference with Mixtures of Isotropic Gaussians
• Authors: Marguerite Petit-Talamon, Marc Lambert, Anna Korba,
• Abstract summary: Variational inference (VI) is a popular approach in Bayesian inference, that looks for the best approximation of the posterior distribution within a parametric family.<n>We develop a variational framework and provide efficient algorithms suited for this family.
• Score: 8.365869192421865
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Variational inference (VI) is a popular approach in Bayesian inference, that looks for the best approximation of the posterior distribution within a parametric family, minimizing a loss that is typically the (reverse) Kullback-Leibler (KL) divergence. In this paper, we focus on the following parametric family: mixtures of isotropic Gaussians (i.e., with diagonal covariance matrices proportional to the identity) and uniform weights. We develop a variational framework and provide efficient algorithms suited for this family. In contrast with mixtures of Gaussian with generic covariance matrices, this choice presents a balance between accurate approximations of multimodal Bayesian posteriors, while being memory and computationally efficient. Our algorithms implement gradient descent on the location of the mixture components (the modes of the Gaussians), and either (an entropic) Mirror or Bures descent on their variance parameters. We illustrate the performance of our algorithms on numerical experiments.

Related papers

• Batch, match, and patch: low-rank approximations for score-based variational inference [8.840147522046651]
  Black-box variational inference scales poorly to high-dimensional problems.<n>We extend the batch-and-match (BaM) framework for score-based BBVI.<n>We evaluate this approach on a variety of synthetic target distributions and real-world problems in high-dimensional inference.
  arXiv  Detail & Related papers  (2024-10-29T17:42:56Z)
• Posterior Covariance Structures in Gaussian Processes [2.1137702137979946]
  We show how the bandwidth parameter and the spatial distribution of the observations influence the posterior covariance.<n>We propose several estimators to efficiently measure the absolute posterior covariance field.<n>We conduct a wide range of experiments to illustrate our theoretical findings and their practical applications.
  arXiv  Detail & Related papers  (2024-08-14T08:56:45Z)
• Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians [27.20127082606962]
  Variational inference (VI) is a popular approach in Bayesian inference. This work aims to contribute to the theoretical study of VI in the non-Gaussian case.
  arXiv  Detail & Related papers  (2024-06-06T12:38:59Z)
• Variational sparse inverse Cholesky approximation for latent Gaussian processes via double Kullback-Leibler minimization [6.012173616364571]
  We combine a variational approximation of the posterior with a similar and efficient SIC-restricted Kullback-Leibler-optimal approximation of the prior. For this setting, our variational approximation can be computed via gradient descent in polylogarithmic time per iteration. We provide numerical comparisons showing that the proposed double-Kullback-Leibler-optimal Gaussian-process approximation (DKLGP) can sometimes be vastly more accurate for stationary kernels than alternative approaches.
  arXiv  Detail & Related papers  (2023-01-30T21:50:08Z)
• Manifold Gaussian Variational Bayes on the Precision Matrix [70.44024861252554]
  We propose an optimization algorithm for Variational Inference (VI) in complex models. We develop an efficient algorithm for Gaussian Variational Inference whose updates satisfy the positive definite constraint on the variational covariance matrix. Due to its black-box nature, MGVBP stands as a ready-to-use solution for VI in complex models.
  arXiv  Detail & Related papers  (2022-10-26T10:12:31Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Theoretical Error Analysis of Entropy Approximation for Gaussian Mixtures [0.6990493129893112]
  In this paper, we study the approximate entropy represented as the sum of the entropies of unimodal Gaussian distributions with mixing coefficients.<n>We theoretically analyze the approximation error between the true and the approximate entropy to reveal when this approximation works effectively.<n>Our results provide a guarantee that this approximation works well for high-dimensional problems, such as neural networks.
  arXiv  Detail & Related papers  (2022-02-26T04:49:01Z)
• A Robust and Flexible EM Algorithm for Mixtures of Elliptical Distributions with Missing Data [71.9573352891936]
  This paper tackles the problem of missing data imputation for noisy and non-Gaussian data. A new EM algorithm is investigated for mixtures of elliptical distributions with the property of handling potential missing data. Experimental results on synthetic data demonstrate that the proposed algorithm is robust to outliers and can be used with non-Gaussian data.
  arXiv  Detail & Related papers  (2022-01-28T10:01:37Z)
• Natural Gradient Variational Inference with Gaussian Mixture Models [1.7948767405202701]
  Variational Inference (VI) methods approximate the posterior with a distribution usually chosen from a simple family using optimization. The main contribution of this work is described is a set of update rules for natural gradient variational inference with mixture of Gaussians.
  arXiv  Detail & Related papers  (2021-11-15T20:04:32Z)
• Scalable Variational Gaussian Processes via Harmonic Kernel Decomposition [54.07797071198249]
  We introduce a new scalable variational Gaussian process approximation which provides a high fidelity approximation while retaining general applicability. We demonstrate that, on a range of regression and classification problems, our approach can exploit input space symmetries such as translations and reflections. Notably, our approach achieves state-of-the-art results on CIFAR-10 among pure GP models.
  arXiv  Detail & Related papers  (2021-06-10T18:17:57Z)
• Spectral clustering under degree heterogeneity: a case for the random walk Laplacian [83.79286663107845]
  This paper shows that graph spectral embedding using the random walk Laplacian produces vector representations which are completely corrected for node degree. In the special case of a degree-corrected block model, the embedding concentrates about K distinct points, representing communities.
  arXiv  Detail & Related papers  (2021-05-03T16:36:27Z)
• Optimal Iterative Sketching with the Subsampled Randomized Hadamard Transform [64.90148466525754]
  We study the performance of iterative sketching for least-squares problems. We show that the convergence rate for Haar and randomized Hadamard matrices are identical, andally improve upon random projections. These techniques may be applied to other algorithms that employ randomized dimension reduction.
  arXiv  Detail & Related papers  (2020-02-03T16:17:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.