Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent

• URL: http://arxiv.org/abs/2305.14076v4
• Date: Sat, 28 Oct 2023 00:37:04 GMT
• Title: Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent
• Authors: Tianle Liu, Promit Ghosal, Krishnakumar Balasubramanian, Natesh S. Pillai
• Abstract summary: Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based deterministic sampling algorithm. We study the dynamics of the Gaussian-SVGD projected to the family of Gaussian distributions via the bilinear kernel. We propose a density-based and a particle-based implementation of the Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from different perspectives, emerge as special cases of our unified framework.
• Score: 16.16051064618816
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Stein Variational Gradient Descent (SVGD) is a nonparametric particle-based deterministic sampling algorithm. Despite its wide usage, understanding the theoretical properties of SVGD has remained a challenging problem. For sampling from a Gaussian target, the SVGD dynamics with a bilinear kernel will remain Gaussian as long as the initializer is Gaussian. Inspired by this fact, we undertake a detailed theoretical study of the Gaussian-SVGD, i.e., SVGD projected to the family of Gaussian distributions via the bilinear kernel, or equivalently Gaussian variational inference (GVI) with SVGD. We present a complete picture by considering both the mean-field PDE and discrete particle systems. When the target is strongly log-concave, the mean-field Gaussian-SVGD dynamics is proven to converge linearly to the Gaussian distribution closest to the target in KL divergence. In the finite-particle setting, there is both uniform in time convergence to the mean-field limit and linear convergence in time to the equilibrium if the target is Gaussian. In the general case, we propose a density-based and a particle-based implementation of the Gaussian-SVGD, and show that several recent algorithms for GVI, proposed from different perspectives, emerge as special cases of our unified framework. Interestingly, one of the new particle-based instance from this framework empirically outperforms existing approaches. Our results make concrete contributions towards obtaining a deeper understanding of both SVGD and GVI.

Related papers

• DiffGS: Functional Gaussian Splatting Diffusion [33.07847512591061]
  3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity. However, the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. We propose DiffGS, a general Gaussian generator based on latent diffusion models.
  arXiv  Detail & Related papers  (2024-10-25T16:08:08Z)
• PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views [116.10577967146762]
  PixelGaussian is an efficient framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Our method achieves state-of-the-art performance with good generalization to various numbers of views.
  arXiv  Detail & Related papers  (2024-10-24T17:59:58Z)
• GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling [55.05713977022407]
  We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. We derive GaussianCube by first using a novel densification-constrained Gaussian fitting algorithm, which yields high-accuracy fitting. Experiments conducted on unconditional and class-conditioned object generation, digital avatar creation, and text-to-3D all show that our model synthesis achieves state-of-the-art generation results.
  arXiv  Detail & Related papers  (2024-03-28T17:59:50Z)
• Analytic-Splatting: Anti-Aliased 3D Gaussian Splatting via Analytic Integration [49.004898985671815]
  3DGS is not alias-free, and its rendering at varying resolutions could produce severe blurring or jaggies. This is because 3DGS treats each pixel as an isolated, single point rather than as an area, causing insensitivity to changes in the footprints of pixels. We introduce this approximation in the two-dimensional pixel shading, and present Analytic-Splatting, which analytically approximates the Gaussian integral within the 2D-pixel window area.
  arXiv  Detail & Related papers  (2024-03-17T02:06:03Z)
• Provably Fast Finite Particle Variants of SVGD via Virtual Particle Stochastic Approximation [9.065034043031668]
  Stein Variational Gradient Descent (SVGD) is a popular variational inference which simulates an interacting particle system to approximately sample from a target distribution. We introduce the notion of virtual particles and develop novel approximations of population-limit dynamics in the space of probability measures. We show that the $n$ particles output by VP-SVGD and GB-SVGD, run for $T$ steps with batch-size $K$, are as good as i.i.i.d samples from a distribution whose Kernel Stein Discrepancy to the target is at most $Oleft(tfrac
  arXiv  Detail & Related papers  (2023-05-27T19:21:28Z)
• Augmented Message Passing Stein Variational Gradient Descent [3.5788754401889014]
  We study the isotropy property of finite particles during the convergence process. All particles tend to cluster around the particle center within a certain range. Our algorithm achieves satisfactory accuracy and overcomes the variance collapse problem in various benchmark problems.
  arXiv  Detail & Related papers  (2023-05-18T01:13:04Z)
• Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalization [81.29406957201458]
  Existing detection methods commonly use a parameterized bounding box (BBox) to model and detect (horizontal) objects. We argue that such a mechanism has fundamental limitations in building an effective regression loss for rotation detection. We propose to model the rotated objects as Gaussian distributions. We extend our approach from 2-D to 3-D with a tailored algorithm design to handle the heading estimation.
  arXiv  Detail & Related papers  (2022-09-22T07:50:48Z)
• Kernel Stein Generative Modeling [68.03537693810972]
  Gradient Langevin Dynamics (SGLD) demonstrates impressive results with energy-based models on high-dimensional and complex data distributions. Stein Variational Gradient Descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate a given distribution. We propose noise conditional kernel SVGD (NCK-SVGD), that works in tandem with the recently introduced Noise Conditional Score Network estimator.
  arXiv  Detail & Related papers  (2020-07-06T21:26:04Z)
• A Non-Asymptotic Analysis for Stein Variational Gradient Descent [44.30569261307296]
  We provide a novel finite time analysis for the Stein Variational Gradient Descent algorithm. We provide a descent lemma establishing that the algorithm decreases the objective at each iteration. We also provide a convergence result of the finite particle system corresponding to the practical implementation of SVGD to its population version.
  arXiv  Detail & Related papers  (2020-06-17T12:01:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.