Variational Inference for Latent Variable Models in High Dimensions

• URL: http://arxiv.org/abs/2506.01893v2
• Date: Wed, 16 Jul 2025 23:11:02 GMT
• Title: Variational Inference for Latent Variable Models in High Dimensions
• Authors: Chenyang Zhong, Sumit Mukherjee, Bodhisattva Sen,
• Abstract summary: We introduce a general framework for quantifying the statistical accuracy of mean-field variational inference (MFVI)<n>We capture the exact regime where MFVI 'works' for the celebrated latent Dirichlet allocation model.<n>Our proof techniques, which extend the framework of nonlinear large deviations, open the door for the analysis of MFVI in other latent variable models.
• Score: 4.3012765978447565
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Variational inference (VI) is a popular method for approximating intractable posterior distributions in Bayesian inference and probabilistic machine learning. In this paper, we introduce a general framework for quantifying the statistical accuracy of mean-field variational inference (MFVI) for posterior approximation in Bayesian latent variable models with categorical local latent variables (and arbitrary global latent variables). Utilizing our general framework, we capture the exact regime where MFVI 'works' for the celebrated latent Dirichlet allocation model. Focusing on the mixed membership stochastic blockmodel, we show that the vanilla fully factorized MFVI, often used in the literature, is suboptimal. We propose a partially grouped VI algorithm for this model and show that it works, and derive its exact finite-sample performance. We further illustrate that our bounds are tight for both the above models. Our proof techniques, which extend the framework of nonlinear large deviations, open the door for the analysis of MFVI in other latent variable models.

Related papers

• Stability of Mean-Field Variational Inference [3.5729687931166136]
  Mean-field inference (MFVI) is a widely used method for approxing high-dimensional probability distributions by product measures.<n>We show that the MFVI depends differentiably on the target potential and characterize the derivative by a partial differential equation.
  arXiv  Detail & Related papers  (2025-06-09T15:21:37Z)
• Bayesian Model Selection via Mean-Field Variational Approximation [10.433170683584994]
  We study the non-asymptotic properties of mean-field (MF) inference under the Bayesian framework. We show a Bernstein von-Mises (BvM) theorem for the variational distribution from MF under possible model mis-specification.
  arXiv  Detail & Related papers  (2023-12-17T04:48:25Z)
• Diffusion models for probabilistic programming [56.47577824219207]
  Diffusion Model Variational Inference (DMVI) is a novel method for automated approximate inference in probabilistic programming languages (PPLs) DMVI is easy to implement, allows hassle-free inference in PPLs without the drawbacks of, e.g., variational inference using normalizing flows, and does not make any constraints on the underlying neural network model.
  arXiv  Detail & Related papers  (2023-11-01T12:17:05Z)
• Bayesian Non-linear Latent Variable Modeling via Random Fourier Features [7.856578780790166]
  We present a method to perform Markov chain Monte Carlo inference for generalized nonlinear latent variable modeling. Inference forVMs is computationally tractable only when the data likelihood is Gaussian. We show that we can generalizeVMs to non-Gaussian observations, such as Poisson, negative binomial, and multinomial distributions.
  arXiv  Detail & Related papers  (2023-06-14T08:42:10Z)
• Manifold Gaussian Variational Bayes on the Precision Matrix [70.44024861252554]
  We propose an optimization algorithm for Variational Inference (VI) in complex models. We develop an efficient algorithm for Gaussian Variational Inference whose updates satisfy the positive definite constraint on the variational covariance matrix. Due to its black-box nature, MGVBP stands as a ready-to-use solution for VI in complex models.
  arXiv  Detail & Related papers  (2022-10-26T10:12:31Z)
• Mean-field Variational Inference via Wasserstein Gradient Flow [8.05603983337769]
  Variational inference, such as the mean-field (MF) approximation, requires certain conjugacy structures for efficient computation. We introduce a general computational framework to implement MFal inference for Bayesian models, with or without latent variables, using the Wasserstein gradient flow (WGF) We propose a new constraint-free function approximation method using neural networks to numerically realize our algorithm.
  arXiv  Detail & Related papers  (2022-07-17T04:05:32Z)
• PAC Generalization via Invariant Representations [41.02828564338047]
  We consider the notion of $epsilon$-approximate invariance in a finite sample setting. Inspired by PAC learning, we obtain finite-sample out-of-distribution generalization guarantees. Our results show bounds that do not scale in ambient dimension when intervention sites are restricted to lie in a constant size subset of in-degree bounded nodes.
  arXiv  Detail & Related papers  (2022-05-30T15:50:14Z)
• Wrapped Distributions on homogeneous Riemannian manifolds [58.720142291102135]
  Control over distributions' properties, such as parameters, symmetry and modality yield a family of flexible distributions. We empirically validate our approach by utilizing our proposed distributions within a variational autoencoder and a latent space network model.
  arXiv  Detail & Related papers  (2022-04-20T21:25:21Z)
• Generalised Gaussian Process Latent Variable Models (GPLVM) with Stochastic Variational Inference [9.468270453795409]
  We study the doubly formulation of the BayesianVM model amenable with minibatch training. We show how this framework is compatible with different latent variable formulations and perform experiments to compare a suite of models. We demonstrate how we can train in the presence of massively missing data and obtain high-fidelity reconstructions.
  arXiv  Detail & Related papers  (2022-02-25T21:21:51Z)
• Efficient CDF Approximations for Normalizing Flows [64.60846767084877]
  We build upon the diffeomorphic properties of normalizing flows to estimate the cumulative distribution function (CDF) over a closed region. Our experiments on popular flow architectures and UCI datasets show a marked improvement in sample efficiency as compared to traditional estimators.
  arXiv  Detail & Related papers  (2022-02-23T06:11:49Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Attentional Prototype Inference for Few-Shot Segmentation [128.45753577331422]
  We propose attentional prototype inference (API), a probabilistic latent variable framework for few-shot segmentation. We define a global latent variable to represent the prototype of each object category, which we model as a probabilistic distribution. We conduct extensive experiments on four benchmarks, where our proposal obtains at least competitive and often better performance than state-of-the-art prototype-based methods.
  arXiv  Detail & Related papers  (2021-05-14T06:58:44Z)
• Posterior Differential Regularization with f-divergence for Improving Model Robustness [95.05725916287376]
  We focus on methods that regularize the model posterior difference between clean and noisy inputs. We generalize the posterior differential regularization to the family of $f$-divergences. Our experiments show that regularizing the posterior differential with $f$-divergence can result in well-improved model robustness.
  arXiv  Detail & Related papers  (2020-10-23T19:58:01Z)
• Variational Filtering with Copula Models for SLAM [5.242618356321224]
  We show how it is possible to perform simultaneous localization and mapping (SLAM) with a larger class of distributions. We integrate the distribution model with copulas into a Sequential Monte Carlo estimator and show how unknown model parameters can be learned through gradient-based optimization.
  arXiv  Detail & Related papers  (2020-08-02T15:38:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.