Projected Stein Variational Gradient Descent

• URL: http://arxiv.org/abs/2002.03469v2
• Date: Wed, 10 Jun 2020 15:00:24 GMT
• Title: Projected Stein Variational Gradient Descent
• Authors: Peng Chen, Omar Ghattas
• Abstract summary: We propose a projected Stein variational descent gradient (pSVGD) method to overcome the challenge of intrinsic low dimensionality of the data informed subspace. We adaptively construct the subspace using a gradient information matrix of the log-likelihood, and apply pSVGD to the much lower-dimensional coefficients of the parameter projection.
• Score: 8.359602391273755
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The curse of dimensionality is a longstanding challenge in Bayesian inference in high dimensions. In this work, we propose a projected Stein variational gradient descent (pSVGD) method to overcome this challenge by exploiting the fundamental property of intrinsic low dimensionality of the data informed subspace stemming from ill-posedness of such problems. We adaptively construct the subspace using a gradient information matrix of the log-likelihood, and apply pSVGD to the much lower-dimensional coefficients of the parameter projection. The method is demonstrated to be more accurate and efficient than SVGD. It is also shown to be more scalable with respect to the number of parameters, samples, data points, and processor cores via experiments with parameters dimensions ranging from the hundreds to the tens of thousands.

Related papers

• Analytic-Splatting: Anti-Aliased 3D Gaussian Splatting via Analytic Integration [49.004898985671815]
  3DGS is not alias-free, and its rendering at varying resolutions could produce severe blurring or jaggies. This is because 3DGS treats each pixel as an isolated, single point rather than as an area, causing insensitivity to changes in the footprints of pixels. We introduce this approximation in the two-dimensional pixel shading, and present Analytic-Splatting, which analytically approximates the Gaussian integral within the 2D-pixel window area.
  arXiv  Detail & Related papers  (2024-03-17T02:06:03Z)
• Augmented Message Passing Stein Variational Gradient Descent [3.5788754401889014]
  We study the isotropy property of finite particles during the convergence process. All particles tend to cluster around the particle center within a certain range. Our algorithm achieves satisfactory accuracy and overcomes the variance collapse problem in various benchmark problems.
  arXiv  Detail & Related papers  (2023-05-18T01:13:04Z)
• Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalization [81.29406957201458]
  Existing detection methods commonly use a parameterized bounding box (BBox) to model and detect (horizontal) objects. We argue that such a mechanism has fundamental limitations in building an effective regression loss for rotation detection. We propose to model the rotated objects as Gaussian distributions. We extend our approach from 2-D to 3-D with a tailored algorithm design to handle the heading estimation.
  arXiv  Detail & Related papers  (2022-09-22T07:50:48Z)
• Grassmann Stein Variational Gradient Descent [3.644031721554146]
  Stein variational gradient descent (SVGD) is a deterministic particle inference algorithm that provides an efficient alternative to Markov chain Monte Carlo. Recent developments have advocated projecting both the score function and the data onto real lines to sidestep this issue. We propose Grassmann Stein variational gradient descent (GSVGD) as an alternative approach, which permits projections onto arbitrary dimensional subspaces.
  arXiv  Detail & Related papers  (2022-02-07T15:36:03Z)
• Projected Wasserstein gradient descent for high-dimensional Bayesian inference [8.750791391857264]
  We propose a projected Wasserstein gradient descent method (pWGD) for high-dimensional Bayesian inference problems. We overcome this challenge by exploiting the intrinsic low-rank structure in the difference between the posterior and prior distributions.
  arXiv  Detail & Related papers  (2021-02-12T05:12:05Z)
• Depth Completion using Piecewise Planar Model [94.0808155168311]
  A depth map can be represented by a set of learned bases and can be efficiently solved in a closed form solution. However, one issue with this method is that it may create artifacts when colour boundaries are inconsistent with depth boundaries. We enforce a more strict model in depth recovery: a piece-wise planar model.
  arXiv  Detail & Related papers  (2020-12-06T07:11:46Z)
• Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs [6.178864935410097]
  We construct surrogates for high-dimensional PDE-governed parametric maps in the form of projected neural networks. We demonstrate that the proposed projected neural network achieves greater accuracy than a full neural network.
  arXiv  Detail & Related papers  (2020-11-30T18:46:40Z)
• Kernel Stein Generative Modeling [68.03537693810972]
  Gradient Langevin Dynamics (SGLD) demonstrates impressive results with energy-based models on high-dimensional and complex data distributions. Stein Variational Gradient Descent (SVGD) is a deterministic sampling algorithm that iteratively transports a set of particles to approximate a given distribution. We propose noise conditional kernel SVGD (NCK-SVGD), that works in tandem with the recently introduced Noise Conditional Score Network estimator.
  arXiv  Detail & Related papers  (2020-07-06T21:26:04Z)
• Sliced Kernelized Stein Discrepancy [17.159499204595527]
  Kernelized Stein discrepancy (KSD) is extensively used in goodness-of-fit tests and model learning. We propose the sliced Stein discrepancy and its scalable and kernelized variants, which employ kernel-based test functions defined on the optimal one-dimensional projections. For model learning, we show its advantages over existing Stein discrepancy baselines by training independent component analysis models with different discrepancies.
  arXiv  Detail & Related papers  (2020-06-30T04:58:55Z)
• Augmented Sliced Wasserstein Distances [55.028065567756066]
  We propose a new family of distance metrics, called augmented sliced Wasserstein distances (ASWDs) ASWDs are constructed by first mapping samples to higher-dimensional hypersurfaces parameterized by neural networks. Numerical results demonstrate that the ASWD significantly outperforms other Wasserstein variants for both synthetic and real-world problems.
  arXiv  Detail & Related papers  (2020-06-15T23:00:08Z)
• Carath\'eodory Sampling for Stochastic Gradient Descent [79.55586575988292]
  We present an approach that is inspired by classical results of Tchakaloff and Carath'eodory about measure reduction. We adaptively select the descent steps where the measure reduction is carried out. We combine this with Block Coordinate Descent so that measure reduction can be done very cheaply.
  arXiv  Detail & Related papers  (2020-06-02T17:52:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.