Semi-Implicit Functional Gradient Flow

• URL: http://arxiv.org/abs/2410.17935v1
• Date: Wed, 23 Oct 2024 15:00:30 GMT
• Title: Semi-Implicit Functional Gradient Flow
• Authors: Shiyue Zhang, Ziheng Cheng, Cheng Zhang,
• Abstract summary: We propose a functional gradient ParVI method that uses perturbed particles as the approximation family. The corresponding functional gradient flow, which can be estimated via denoising score matching, exhibits strong theoretical convergence guarantee.
• Score: 30.32233517392456
• License:
• Abstract: Particle-based variational inference methods (ParVIs) use non-parametric variational families represented by particles to approximate the target distribution according to the kernelized Wasserstein gradient flow for the Kullback-Leibler (KL) divergence. Recent works introduce functional gradient flows to substitute the kernel for better flexibility. However, the deterministic updating mechanism may suffer from limited exploration and require expensive repetitive runs for new samples. In this paper, we propose Semi-Implicit Functional Gradient flow (SIFG), a functional gradient ParVI method that uses perturbed particles as the approximation family. The corresponding functional gradient flow, which can be estimated via denoising score matching, exhibits strong theoretical convergence guarantee. We also present an adaptive version of our method to automatically choose the suitable noise magnitude. Extensive experiments demonstrate the effectiveness and efficiency of the proposed framework on both simulated and real data problems.

Related papers

• Functional Gradient Flows for Constrained Sampling [29.631753643887237]
  We propose a new functional gradient ParVI method for constrained sampling, called constrained functional gradient flow (CFG) We also present novel numerical strategies to handle the boundary integral term arising from the domain constraints.
  arXiv  Detail & Related papers  (2024-10-30T16:20:48Z)
• Conditional Lagrangian Wasserstein Flow for Time Series Imputation [3.914746375834628]
  We propose a novel method for time series imputation called Conditional Lagrangian Wasserstein Flow. The proposed method leverages the (conditional) optimal transport theory to learn the probability flow in a simulation-free manner. The experimental results on the real-word datasets show that the proposed method achieves competitive performance on time series imputation.
  arXiv  Detail & Related papers  (2024-10-10T02:46:28Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Particle-based Variational Inference with Generalized Wasserstein Gradient Flow [32.37056212527921]
  We propose a ParVI framework, called generalized Wasserstein gradient descent (GWG) We show that GWG exhibits strong convergence guarantees. We also provide an adaptive version that automatically chooses Wasserstein metric to accelerate convergence.
  arXiv  Detail & Related papers  (2023-10-25T10:05:42Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Particle-based Variational Inference with Preconditioned Functional Gradient Flow [13.519223374081648]
  We propose a new particle-based variational inference algorithm called preconditioned functional gradient flow (PFG) PFG has several advantages over Stein variational gradient descent (SVGD) Non-linear function classes such as neural networks can be incorporated to estimate the gradient flow.
  arXiv  Detail & Related papers  (2022-11-25T08:31:57Z)
• Sampling with Mollified Interaction Energy Descent [57.00583139477843]
  We present a new optimization-based method for sampling called mollified interaction energy descent (MIED) MIED minimizes a new class of energies on probability measures called mollified interaction energies (MIEs) We show experimentally that for unconstrained sampling problems our algorithm performs on par with existing particle-based algorithms like SVGD.
  arXiv  Detail & Related papers  (2022-10-24T16:54:18Z)
• A blob method method for inhomogeneous diffusion with applications to multi-agent control and sampling [0.6562256987706128]
  We develop a deterministic particle method for the weighted porous medium equation (WPME) and prove its convergence on bounded time intervals. Our method has natural applications to multi-agent coverage algorithms and sampling probability measures.
  arXiv  Detail & Related papers  (2022-02-25T19:49:05Z)
• Differentiable Annealed Importance Sampling and the Perils of Gradient Noise [68.44523807580438]
  Annealed importance sampling (AIS) and related algorithms are highly effective tools for marginal likelihood estimation. Differentiability is a desirable property as it would admit the possibility of optimizing marginal likelihood as an objective. We propose a differentiable algorithm by abandoning Metropolis-Hastings steps, which further unlocks mini-batch computation.
  arXiv  Detail & Related papers  (2021-07-21T17:10:14Z)
• Variational Transport: A Convergent Particle-BasedAlgorithm for Distributional Optimization [106.70006655990176]
  A distributional optimization problem arises widely in machine learning and statistics. We propose a novel particle-based algorithm, dubbed as variational transport, which approximately performs Wasserstein gradient descent. We prove that when the objective function satisfies a functional version of the Polyak-Lojasiewicz (PL) (Polyak, 1963) and smoothness conditions, variational transport converges linearly.
  arXiv  Detail & Related papers  (2020-12-21T18:33:13Z)
• SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives [86.01677297601624]
  We propose a novel approach for scaling GP regression with derivatives based on quadrature Fourier features. We prove deterministic, non-asymptotic and exponentially fast decaying error bounds which apply for both the approximated kernel as well as the approximated posterior.
  arXiv  Detail & Related papers  (2020-03-05T14:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.