Langevin Diffusion Variational Inference

• URL: http://arxiv.org/abs/2208.07743v2
• Date: Thu, 23 Mar 2023 13:54:52 GMT
• Title: Langevin Diffusion Variational Inference
• Authors: Tomas Geffner and Justin Domke
• Abstract summary: We give a single analysis that unifies and generalizes existing techniques. The main idea is to augment the target and variational by numerically simulating the underdamped Langevin diffusion process. We propose a new method that combines the strengths of previously existing algorithms.
• Score: 38.73307745906571
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many methods that build powerful variational distributions based on unadjusted Langevin transitions exist. Most of these were developed using a wide range of different approaches and techniques. Unfortunately, the lack of a unified analysis and derivation makes developing new methods and reasoning about existing ones a challenging task. We address this giving a single analysis that unifies and generalizes these existing techniques. The main idea is to augment the target and variational by numerically simulating the underdamped Langevin diffusion process and its time reversal. The benefits of this approach are twofold: it provides a unified formulation for many existing methods, and it simplifies the development of new ones. In fact, using our formulation we propose a new method that combines the strengths of previously existing algorithms; it uses underdamped Langevin transitions and powerful augmentations parameterized by a score network. Our empirical evaluation shows that our proposed method consistently outperforms relevant baselines in a wide range of tasks.

Related papers

• Conditioning Diffusions Using Malliavin Calculus [18.62300657866048]
  In generative modelling and optimal control, a central computational task is to modify a reference diffusion process to maximise a given terminal-time reward.<n>We introduce a novel framework based on Malliavin calculus and centred around a generalisation of the Tweedie score formula to nonlinear differential equations.<n>This allows our approach to handle a broad range of applications, like diffusion bridges, or adding conditional controls to an already trained diffusion model.
  arXiv  Detail & Related papers  (2025-04-04T14:10:21Z)
• Variational empirical Bayes variable selection in high-dimensional logistic regression [2.4032899110671955]
  We develop a novel and computationally efficient variational approximation thereof. One such novelty is that we develop this approximation directly for the marginal distribution on the model space, rather than on the regression coefficients themselves. We demonstrate the method's strong performance in simulations, and prove that our variational approximation inherits the strong selection consistency property satisfied by the posterior distribution that it is approximating.
  arXiv  Detail & Related papers  (2025-02-14T19:57:13Z)
• Diffusing Differentiable Representations [60.72992910766525]
  We introduce a novel, training-free method for sampling differentiable representations (diffreps) using pretrained diffusion models. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects.
  arXiv  Detail & Related papers  (2024-12-09T20:42:58Z)
• Adversarial Schrödinger Bridge Matching [66.39774923893103]
  Iterative Markovian Fitting (IMF) procedure alternates between Markovian and reciprocal projections of continuous-time processes. We propose a novel Discrete-time IMF (D-IMF) procedure in which learning of processes is replaced by learning just a few transition probabilities in discrete time. We show that our D-IMF procedure can provide the same quality of unpaired domain translation as the IMF, using only several generation steps instead of hundreds.
  arXiv  Detail & Related papers  (2024-05-23T11:29:33Z)
• ODE Discovery for Longitudinal Heterogeneous Treatment Effects Inference [69.24516189971929]
  In this paper, we introduce a new type of solution in the longitudinal setting: a closed-form ordinary differential equation (ODE) While we still rely on continuous optimization to learn an ODE, the resulting inference machine is no longer a neural network.
  arXiv  Detail & Related papers  (2024-03-16T02:07:45Z)
• Implicit Variational Inference for High-Dimensional Posteriors [7.924706533725115]
  In variational inference, the benefits of Bayesian models rely on accurately capturing the true posterior distribution. We propose using neural samplers that specify implicit distributions, which are well-suited for approximating complex multimodal and correlated posteriors. Our approach introduces novel bounds for approximate inference using implicit distributions by locally linearising the neural sampler.
  arXiv  Detail & Related papers  (2023-10-10T14:06:56Z)
• Improved sampling via learned diffusions [8.916420423563478]
  Recently, a series of papers proposed deep learning-based approaches to sample from target distributions using controlled diffusion processes. We identify these approaches as special cases of a generalized Schr"odinger bridge problem. We propose a variational formulation based on divergences between path space measures of time-reversed diffusion processes.
  arXiv  Detail & Related papers  (2023-07-03T17:58:26Z)
• Sublinear Convergence Rates of Extragradient-Type Methods: A Survey on Classical and Recent Developments [12.995632804090198]
  The extragradient (EG) is a well-known method to approximate solutions of saddle-point problems. Recently, these methods have gained popularity due to new applications in machine learning and robust optimization. We provide a unified convergence analysis for different classes of algorithms, with an emphasis on sublinear best-iterate and last-iterate convergence rates.
  arXiv  Detail & Related papers  (2023-03-30T07:04:22Z)
• Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examples [89.85593878754571]
  transferability of adversarial examples across deep neural networks is the crux of many black-box attacks. We advocate to attack a Bayesian model for achieving desirable transferability. Our method outperforms recent state-of-the-arts by large margins.
  arXiv  Detail & Related papers  (2023-02-10T07:08:13Z)
• Improving Diffusion Models for Inverse Problems using Manifold Constraints [55.91148172752894]
  We show that current solvers throw the sample path off the data manifold, and hence the error accumulates. To address this, we propose an additional correction term inspired by the manifold constraint. We show that our method is superior to the previous methods both theoretically and empirically.
  arXiv  Detail & Related papers  (2022-06-02T09:06:10Z)
• Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods [73.35353358543507]
  Gradient Descent-Ascent (SGDA) is one of the most prominent algorithms for solving min-max optimization and variational inequalities problems (VIP) In this paper, we propose a unified convergence analysis that covers a large variety of descent-ascent methods. We develop several new variants of SGDA such as a new variance-reduced method (L-SVRGDA), new distributed methods with compression (QSGDA, DIANA-SGDA, VR-DIANA-SGDA), and a new method with coordinate randomization (SEGA-SGDA)
  arXiv  Detail & Related papers  (2022-02-15T09:17:39Z)
• Simulating Diffusion Bridges with Score Matching [35.28393521329226]
  We consider the problem of simulating diffusion bridges, which are diffusion processes that are conditioned to initialize and terminate at two given states.<n>This article contributes to this rich body of literature by presenting a new avenue to obtain diffusion bridge approximations.
  arXiv  Detail & Related papers  (2021-11-14T05:18:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.