Online, Informative MCMC Thinning with Kernelized Stein Discrepancy

• URL: http://arxiv.org/abs/2201.07130v1
• Date: Tue, 18 Jan 2022 17:13:21 GMT
• Title: Online, Informative MCMC Thinning with Kernelized Stein Discrepancy
• Authors: Cole Hawkins, Alec Koppel, Zheng Zhang
• Abstract summary: We propose an MCMC variant that retains only those posterior samples which exceed a KSD threshold. We establish the convergence and complexity tradeoffs for several settings of KSD Thinning. Code is available atcolehawkins.com/colehawkins/KSD-Thinning.
• Score: 14.218897034491125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A fundamental challenge in Bayesian inference is efficient representation of a target distribution. Many non-parametric approaches do so by sampling a large number of points using variants of Markov Chain Monte Carlo (MCMC). We propose an MCMC variant that retains only those posterior samples which exceed a KSD threshold, which we call KSD Thinning. We establish the convergence and complexity tradeoffs for several settings of KSD Thinning as a function of the KSD threshold parameter, sample size, and other problem parameters. Finally, we provide experimental comparisons against other online nonparametric Bayesian methods that generate low-complexity posterior representations, and observe superior consistency/complexity tradeoffs. Code is available at github.com/colehawkins/KSD-Thinning.

Related papers

• Sinkhorn Distance Minimization for Knowledge Distillation [97.64216712016571]
  Knowledge distillation (KD) has been widely adopted to compress large language models (LLMs) In this paper, we show that the aforementioned KL, RKL, and JS divergences respectively suffer from issues of mode-averaging, mode-collapsing, and mode-underestimation. We propose the Sinkhorn Knowledge Distillation (SinKD) that exploits the Sinkhorn distance to ensure a nuanced and precise assessment of the disparity between teacher and student distributions.
  arXiv  Detail & Related papers  (2024-02-27T01:13:58Z)
• Bayesian Decision Trees Inspired from Evolutionary Algorithms [64.80360020499555]
  We propose a replacement of the Markov Chain Monte Carlo (MCMC) with an inherently parallel algorithm, the Sequential Monte Carlo (SMC) Experiments show that SMC combined with the Evolutionary Algorithms (EA) can produce more accurate results compared to MCMC in 100 times fewer iterations.
  arXiv  Detail & Related papers  (2023-05-30T06:17:35Z)
• Bayesian Pseudo-Coresets via Contrastive Divergence [5.479797073162603]
  We introduce a novel approach for constructing pseudo-coresets by utilizing contrastive divergence. It eliminates the need for approximations in the pseudo-coreset construction process. We conduct extensive experiments on multiple datasets, demonstrating its superiority over existing BPC techniques.
  arXiv  Detail & Related papers  (2023-03-20T17:13:50Z)
• Preconditioned Score-based Generative Models [49.88840603798831]
  An intuitive acceleration method is to reduce the sampling iterations which however causes severe performance degradation. We propose a model-agnostic bfem preconditioned diffusion sampling (PDS) method that leverages matrix preconditioning to alleviate the aforementioned problem. PDS alters the sampling process of a vanilla SGM at marginal extra computation cost, and without model retraining.
  arXiv  Detail & Related papers  (2023-02-13T16:30:53Z)
• Controlling Moments with Kernel Stein Discrepancies [74.82363458321939]
  Kernel Stein discrepancies (KSDs) measure the quality of a distributional approximation. We first show that standard KSDs used for weak convergence control fail to control moment convergence. We then provide sufficient conditions under which alternative diffusion KSDs control both moment and weak convergence.
  arXiv  Detail & Related papers  (2022-11-10T08:24:52Z)
• Online Probabilistic Model Identification using Adaptive Recursive MCMC [8.465242072268019]
  We suggest the Adaptive Recursive Markov Chain Monte Carlo (ARMCMC) method. It eliminates the shortcomings of conventional online techniques while computing the entire probability density function of model parameters. We demonstrate our approach using parameter estimation in a soft bending actuator and the Hunt-Crossley dynamic model.
  arXiv  Detail & Related papers  (2022-10-23T02:06:48Z)
• Targeted Separation and Convergence with Kernel Discrepancies [61.973643031360254]
  kernel-based discrepancy measures are required to (i) separate a target P from other probability measures or (ii) control weak convergence to P. In this article we derive new sufficient and necessary conditions to ensure (i) and (ii) For MMDs on separable metric spaces, we characterize those kernels that separate Bochner embeddable measures and introduce simple conditions for separating all measures with unbounded kernels.
  arXiv  Detail & Related papers  (2022-09-26T16:41:16Z)
• MCMC Variational Inference via Uncorrected Hamiltonian Annealing [42.26118870861363]
  We propose a framework to use an AIS-like procedure with Uncorrected Hamiltonian MCMC. Our method leads to tight and differentiable lower bounds on log Z.
  arXiv  Detail & Related papers  (2021-07-08T23:59:45Z)
• Stochastic Gradient MCMC with Multi-Armed Bandit Tuning [2.2559617939136505]
  We propose a novel bandit-based algorithm that tunes SGMCMC hyperparameters to maximize the accuracy of the posterior approximation. We support our results with experiments on both simulated and real datasets, and find that this method is practical for a wide range of application areas.
  arXiv  Detail & Related papers  (2021-05-27T11:00:31Z)
• What Are Bayesian Neural Network Posteriors Really Like? [63.950151520585024]
  We show that Hamiltonian Monte Carlo can achieve significant performance gains over standard and deep ensembles. We also show that deep distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.
  arXiv  Detail & Related papers  (2021-04-29T15:38:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.