Accelerating Stochastic Probabilistic Inference

• URL: http://arxiv.org/abs/2203.07585v1
• Date: Tue, 15 Mar 2022 01:19:12 GMT
• Title: Accelerating Stochastic Probabilistic Inference
• Authors: Minta Liu, Suliang Bu
• Abstract summary: Variational Inference (SVI) has been increasingly attractive thanks to its ability to find good posterior approximations of probabilistic models. Almost all the state-of-the-art SVI algorithms are based on first-order optimization and often suffer from poor convergence rate. We bridge the gap between second-order methods and variational inference by proposing a second-order based variational inference approach.
• Score: 1.599072005190786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Stochastic Variational Inference (SVI) has been increasingly attractive thanks to its ability to find good posterior approximations of probabilistic models. It optimizes the variational objective with stochastic optimization, following noisy estimates of the natural gradient. However, almost all the state-of-the-art SVI algorithms are based on first-order optimization algorithm and often suffer from poor convergence rate. In this paper, we bridge the gap between second-order methods and stochastic variational inference by proposing a second-order based stochastic variational inference approach. In particular, firstly we derive the Hessian matrix of the variational objective. Then we devise two numerical schemes to implement second-order SVI efficiently. Thorough empirical evaluations are investigated on both synthetic and real dataset to backup both the effectiveness and efficiency of the proposed approach.

Related papers

• Variational Autoencoders for Efficient Simulation-Based Inference [0.3495246564946556]
  We present a generative modeling approach based on the variational inference framework for likelihood-free simulation-based inference. We demonstrate the efficacy of these models on well-established benchmark problems, achieving results comparable to flow-based approaches.
  arXiv  Detail & Related papers  (2024-11-21T12:24:13Z)
• Bayesian Optimization for Non-Convex Two-Stage Stochastic Optimization Problems [2.9016548477524156]
  We formulate a knowledgeient-based acquisition function to jointly optimize the first and second-stage variables. We show that differences in the dimension and length scales between the variable types can lead to inefficiencies of the twostep algorithm.
  arXiv  Detail & Related papers  (2024-08-30T16:26:31Z)
• Learning Unnormalized Statistical Models via Compositional Optimization [73.30514599338407]
  Noise-contrastive estimation(NCE) has been proposed by formulating the objective as the logistic loss of the real data and the artificial noise. In this paper, we study it a direct approach for optimizing the negative log-likelihood of unnormalized models.
  arXiv  Detail & Related papers  (2023-06-13T01:18:16Z)
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization [80.8226518642952]
  We study optimization of areas under precision-recall curves (AUPRC), which is widely used for imbalanced tasks. We develop novel momentum methods with a better iteration of $O (1/epsilon4)$ for finding an $epsilon$stationary solution. We also design a novel family of adaptive methods with the same complexity of $O (1/epsilon4)$, which enjoy faster convergence in practice.
  arXiv  Detail & Related papers  (2021-07-02T16:21:52Z)
• High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise [51.31435087414348]
  It is essential to theoretically guarantee that algorithms provide small objective residual with high probability. Existing methods for non-smooth convex optimization have complexity bounds with dependence on confidence level. We propose novel stepsize rules for two methods with gradient clipping.
  arXiv  Detail & Related papers  (2021-06-10T17:54:21Z)
• Stochastic Learning Approach to Binary Optimization for Optimal Design of Experiments [0.0]
  We present a novel approach to binary optimization for optimal experimental design (OED) for Bayesian inverse problems governed by mathematical models such as partial differential equations. The OED utility function, namely, the regularized optimality gradient, is cast into an objective function in the form of an expectation over a Bernoulli distribution. The objective is then solved by using a probabilistic optimization routine to find an optimal observational policy.
  arXiv  Detail & Related papers  (2021-01-15T03:54:12Z)
• Zeroth-Order Hybrid Gradient Descent: Towards A Principled Black-Box Optimization Framework [100.36569795440889]
  This work is on the iteration of zero-th-order (ZO) optimization which does not require first-order information. We show that with a graceful design in coordinate importance sampling, the proposed ZO optimization method is efficient both in terms of complexity as well as as function query cost.
  arXiv  Detail & Related papers  (2020-12-21T17:29:58Z)
• Bilevel Optimization: Convergence Analysis and Enhanced Design [63.64636047748605]
  Bilevel optimization is a tool for many machine learning problems. We propose a novel stoc-efficientgradient estimator named stoc-BiO.
  arXiv  Detail & Related papers  (2020-10-15T18:09:48Z)
• On the implementation of a global optimization method for mixed-variable problems [0.30458514384586394]
  The algorithm is based on the radial basis function of Gutmann and the metric response surface method of Regis and Shoemaker. We propose several modifications aimed at generalizing and improving these two algorithms.
  arXiv  Detail & Related papers  (2020-09-04T13:36:56Z)
• Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
  We show that common optimization methods lead to poor variational approximations if the problem is moderately large. Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
  arXiv  Detail & Related papers  (2020-09-01T19:12:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.