Accelerating Convergence of Stein Variational Gradient Descent via Deep Unfolding

• URL: http://arxiv.org/abs/2402.15125v1
• Date: Fri, 23 Feb 2024 06:24:57 GMT
• Title: Accelerating Convergence of Stein Variational Gradient Descent via Deep Unfolding
• Authors: Yuya Kawamura and Satoshi Takabe
• Abstract summary: Stein variational gradient descent (SVGD) is a prominent particle-based variational inference method used for sampling a target distribution. In this paper, we propose novel trainable algorithms that incorporate a deep-learning technique called deep unfolding,into SVGD.
• Score: 5.584060970507506
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stein variational gradient descent (SVGD) is a prominent particle-based variational inference method used for sampling a target distribution. SVGD has attracted interest for application in machine-learning techniques such as Bayesian inference. In this paper, we propose novel trainable algorithms that incorporate a deep-learning technique called deep unfolding,into SVGD. This approach facilitates the learning of the internal parameters of SVGD, thereby accelerating its convergence speed. To evaluate the proposed trainable SVGD algorithms, we conducted numerical simulations of three tasks: sampling a one-dimensional Gaussian mixture, performing Bayesian logistic regression, and learning Bayesian neural networks. The results show that our proposed algorithms exhibit faster convergence than the conventional variants of SVGD.

Related papers

• A forward differential deep learning-based algorithm for solving high-dimensional nonlinear backward stochastic differential equations [0.6040014326756179]
  We present a novel forward differential deep learning-based algorithm for solving high-dimensional nonlinear backward differential equations (BSDEs) Motivated by the fact that differential deep learning can efficiently approximate the labels and their derivatives with respect to inputs, we transform the BSDE problem into a differential deep learning problem. The main idea of our algorithm is to discretize the integrals using the Euler-Maruyama method and approximate the unknown discrete solution triple using three deep neural networks.
  arXiv  Detail & Related papers  (2024-08-10T19:34:03Z)
• R$^2$-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction [53.19869886963333]
  3D Gaussian splatting (3DGS) has shown promising results in rendering image and surface reconstruction. This paper introduces R2$-Gaussian, the first 3DGS-based framework for sparse-view tomographic reconstruction.
  arXiv  Detail & Related papers  (2024-05-31T08:39:02Z)
• Ensemble Quadratic Assignment Network for Graph Matching [52.20001802006391]
  Graph matching is a commonly used technique in computer vision and pattern recognition. Recent data-driven approaches have improved the graph matching accuracy remarkably. We propose a graph neural network (GNN) based approach to combine the advantages of data-driven and traditional methods.
  arXiv  Detail & Related papers  (2024-03-11T06:34:05Z)
• Geometry-Informed Neural Operator for Large-Scale 3D PDEs [76.06115572844882]
  We propose the geometry-informed neural operator (GINO) to learn the solution operator of large-scale partial differential equations. We successfully trained GINO to predict the pressure on car surfaces using only five hundred data points.
  arXiv  Detail & Related papers  (2023-09-01T16:59:21Z)
• Augmented Message Passing Stein Variational Gradient Descent [3.5788754401889014]
  We study the isotropy property of finite particles during the convergence process. All particles tend to cluster around the particle center within a certain range. Our algorithm achieves satisfactory accuracy and overcomes the variance collapse problem in various benchmark problems.
  arXiv  Detail & Related papers  (2023-05-18T01:13:04Z)
• Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalization [81.29406957201458]
  Existing detection methods commonly use a parameterized bounding box (BBox) to model and detect (horizontal) objects. We argue that such a mechanism has fundamental limitations in building an effective regression loss for rotation detection. We propose to model the rotated objects as Gaussian distributions. We extend our approach from 2-D to 3-D with a tailored algorithm design to handle the heading estimation.
  arXiv  Detail & Related papers  (2022-09-22T07:50:48Z)
• A stochastic Stein Variational Newton method [7.272730677575111]
  We show that Stein variational Newton (sSVN) is a promising approach to accelerating high-precision Bayesian inference tasks. We demonstrate the effectiveness of our algorithm on a difficult class of test problems -- the Hybrid Rosenbrock density -- and show that sSVN converges using three orders of fewer magnitude evaluations of the log likelihood.
  arXiv  Detail & Related papers  (2022-04-19T17:57:36Z)
• Grassmann Stein Variational Gradient Descent [3.644031721554146]
  Stein variational gradient descent (SVGD) is a deterministic particle inference algorithm that provides an efficient alternative to Markov chain Monte Carlo. Recent developments have advocated projecting both the score function and the data onto real lines to sidestep this issue. We propose Grassmann Stein variational gradient descent (GSVGD) as an alternative approach, which permits projections onto arbitrary dimensional subspaces.
  arXiv  Detail & Related papers  (2022-02-07T15:36:03Z)
• A Differentiable Point Process with Its Application to Spiking Neural Networks [13.160616423673373]
  Jimenez Rezende & Gerstner (2014) proposed a variational inference algorithm to train SNNs with hidden neurons. This paper presents an alternative gradient estimator for SNNs based on the path-wise gradient estimator.
  arXiv  Detail & Related papers  (2021-06-02T02:40:17Z)
• Kernel Stein Generative Modeling [68.03537693810972]
  Gradient Langevin Dynamics (SGLD) demonstrates impressive results with energy-based models on high-dimensional and complex data distributions. Stein Variational Gradient Descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate a given distribution. We propose noise conditional kernel SVGD (NCK-SVGD), that works in tandem with the recently introduced Noise Conditional Score Network estimator.
  arXiv  Detail & Related papers  (2020-07-06T21:26:04Z)
• Stein Variational Inference for Discrete Distributions [70.19352762933259]
  We propose a simple yet general framework that transforms discrete distributions to equivalent piecewise continuous distributions. Our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN) In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions.
  arXiv  Detail & Related papers  (2020-03-01T22:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.