Towards understanding Accelerated Stein Variational Gradient Flow -- Analysis of Generalized Bilinear Kernels for Gaussian target distributions

• URL: http://arxiv.org/abs/2509.04008v2
• Date: Wed, 01 Oct 2025 06:06:25 GMT
• Title: Towards understanding Accelerated Stein Variational Gradient Flow -- Analysis of Generalized Bilinear Kernels for Gaussian target distributions
• Authors: Viktor Stein, Wuchen Li,
• Abstract summary: Stein variational gradient descent (SVGD) is a kernel-based and non-parametric particle method for sampling from a target distribution.<n>We introduce accelerated SVGD (ASVGD), based on an accelerated gradient flow in a metric space of probability densities.<n>In the setting of Bayesian neural networks, ASVGD outperforms SVGD significantly in terms of log-likelihood and total times.
• Score: 0.5528896840956629
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stein variational gradient descent (SVGD) is a kernel-based and non-parametric particle method for sampling from a target distribution, such as in Bayesian inference and other machine learning tasks. Different from other particle methods, SVGD does not require estimating the score, which is the gradient of the log-density. However, in practice, SVGD can be slow compared to score-estimation-based sampling algorithms. To design a fast and efficient high-dimensional sampling algorithm with the advantages of SVGD, we introduce accelerated SVGD (ASVGD), based on an accelerated gradient flow in a metric space of probability densities following Nesterov's method. We then derive a momentum-based discrete-time sampling algorithm, which evolves a set of particles deterministically. To stabilize the particles' position update, we also include a Wasserstein metric regularization. This paper extends the conference version \cite{SL2025}. For the bilinear kernel and Gaussian target distributions, we study the kernel parameter and damping parameters with an optimal convergence rate of the proposed dynamics. This is achieved by analyzing the linearized accelerated gradient flows at the equilibrium. Interestingly, the optimal parameter is a constant, which does not depend on the covariance of the target distribution. For the generalized kernel functions, such as the Gaussian kernel, numerical examples with varied target distributions demonstrate the effectiveness of ASVGD compared to SVGD and other popular sampling methods. Furthermore, we show that in the setting of Bayesian neural networks, ASVGD outperforms SVGD significantly in terms of log-likelihood and total iteration times.

Related papers

• Accelerated Stein Variational Gradient Flow [2.384873896423002]
  Stein variational gradient descent (SVGD) is a kernel-based particle method for sampling from a target distribution.<n>We introduce ASVGD, an accelerated SVGD, based on an accelerated gradient flow in a metric space of probability densities.<n>We derive a momentum-based discrete-time sampling algorithm, which evolves a set of particles deterministically.
  arXiv  Detail & Related papers  (2025-03-30T14:37:21Z)
• Accelerating Convergence of Stein Variational Gradient Descent via Deep Unfolding [5.584060970507506]
  Stein variational gradient descent (SVGD) is a prominent particle-based variational inference method used for sampling a target distribution. In this paper, we propose novel trainable algorithms that incorporate a deep-learning technique called deep unfolding,into SVGD.
  arXiv  Detail & Related papers  (2024-02-23T06:24:57Z)
• Provably Fast Finite Particle Variants of SVGD via Virtual Particle Stochastic Approximation [9.065034043031668]
  Stein Variational Gradient Descent (SVGD) is a popular variational inference which simulates an interacting particle system to approximately sample from a target distribution. We introduce the notion of virtual particles and develop novel approximations of population-limit dynamics in the space of probability measures. We show that the $n$ particles output by VP-SVGD and GB-SVGD, run for $T$ steps with batch-size $K$, are as good as i.i.i.d samples from a distribution whose Kernel Stein Discrepancy to the target is at most $Oleft(tfrac
  arXiv  Detail & Related papers  (2023-05-27T19:21:28Z)
• Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent [16.16051064618816]
  Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based deterministic sampling algorithm. We study the dynamics of the Gaussian-SVGD projected to the family of Gaussian distributions via the bilinear kernel. We propose a density-based and a particle-based implementation of the Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from different perspectives, emerge as special cases of our unified framework.
  arXiv  Detail & Related papers  (2023-05-23T13:55:47Z)
• A Finite-Particle Convergence Rate for Stein Variational Gradient Descent [47.6818454221125]
  We provide the first finite-particle convergence rate for Stein variational descent gradient (SVGD) Our explicit, non-asymptotic proof strategy will serve as a template for future refinements.
  arXiv  Detail & Related papers  (2022-11-17T17:50:39Z)
• Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalization [81.29406957201458]
  Existing detection methods commonly use a parameterized bounding box (BBox) to model and detect (horizontal) objects. We argue that such a mechanism has fundamental limitations in building an effective regression loss for rotation detection. We propose to model the rotated objects as Gaussian distributions. We extend our approach from 2-D to 3-D with a tailored algorithm design to handle the heading estimation.
  arXiv  Detail & Related papers  (2022-09-22T07:50:48Z)
• Large-Scale Wasserstein Gradient Flows [84.73670288608025]
  We introduce a scalable scheme to approximate Wasserstein gradient flows. Our approach relies on input neural networks (ICNNs) to discretize the JKO steps. As a result, we can sample from the measure at each step of the gradient diffusion and compute its density.
  arXiv  Detail & Related papers  (2021-06-01T19:21:48Z)
• Kernel Stein Generative Modeling [68.03537693810972]
  Gradient Langevin Dynamics (SGLD) demonstrates impressive results with energy-based models on high-dimensional and complex data distributions. Stein Variational Gradient Descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate a given distribution. We propose noise conditional kernel SVGD (NCK-SVGD), that works in tandem with the recently introduced Noise Conditional Score Network estimator.
  arXiv  Detail & Related papers  (2020-07-06T21:26:04Z)
• A Non-Asymptotic Analysis for Stein Variational Gradient Descent [44.30569261307296]
  We provide a novel finite time analysis for the Stein Variational Gradient Descent algorithm. We provide a descent lemma establishing that the algorithm decreases the objective at each iteration. We also provide a convergence result of the finite particle system corresponding to the practical implementation of SVGD to its population version.
  arXiv  Detail & Related papers  (2020-06-17T12:01:33Z)
• Stein Variational Inference for Discrete Distributions [70.19352762933259]
  We propose a simple yet general framework that transforms discrete distributions to equivalent piecewise continuous distributions. Our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN) In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions.
  arXiv  Detail & Related papers  (2020-03-01T22:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.