Related papers: Gaussian Process Tilted Nonparametric Density Estimation using Fisher Divergence Score Matching

Gaussian Process Tilted Nonparametric Density Estimation using Fisher Divergence Score Matching

URL: http://arxiv.org/abs/2504.03485v1
Date: Fri, 04 Apr 2025 14:41:41 GMT
Title: Gaussian Process Tilted Nonparametric Density Estimation using Fisher Divergence Score Matching
Authors: John Paisley, Wei Zhang, Brian Barr,
Abstract summary: We present three Fisher divergence (FD) algorithms for learning Gaussian process (GP) based score models.<n>We show that all learning problems can be solved in closed form.<n>We demonstrate our three learning algorithms, as well as a MAP baseline algorithm, on several low dimensional density estimation problems.
Score: 3.7969209746164325
License: http://creativecommons.org/licenses/by/4.0/
Abstract: We present three Fisher divergence (FD) minimization algorithms for learning Gaussian process (GP) based score models for lower dimensional density estimation problems. The density is formed by multiplying a base multivariate normal distribution with an exponentiated GP refinement, and so we refer to it as a GP-tilted nonparametric density. By representing the GP part of the score as a linear function using the random Fourier feature (RFF) approximation, we show that all learning problems can be solved in closed form. This includes the basic and noise conditional versions of the Fisher divergence, as well as a novel alternative to noise conditional FD models based on variational inference (VI). Here, we propose using an ELBO-like optimization of the approximate posterior with which we derive a Fisher variational predictive distribution. The RFF representation of the GP, which is functionally equivalent to a single layer neural network score model with cosine activation, provides a unique linear form for which all expectations are in closed form. The Gaussian base also helps with tractability of the VI approximation. We demonstrate our three learning algorithms, as well as a MAP baseline algorithm, on several low dimensional density estimation problems. The closed-form nature of the learning problem removes the reliance on iterative algorithms, making this technique particularly well-suited to large data sets.

Related papers

GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning [52.295563400314094]
Second-order methods are widely adopted to improve the convergence rate of learning algorithms. This paper introduces a novel second-order FL framework tailored for wireless channels.
arXiv Detail & Related papers (2024-12-05T04:27:41Z)
Graph Laplacian-based Bayesian Multi-fidelity Modeling [1.383698759122035]
A graph Laplacian constructed from the low-fidelity data is used to define a multivariate Gaussian prior density. Few high-fidelity data points are used to construct a conjugate likelihood term. The results demonstrate that by utilizing a small fraction of high-fidelity data, the multi-fidelity approach can significantly improve the accuracy of a large collection of low-fidelity data points.
arXiv Detail & Related papers (2024-09-12T16:51:55Z)
Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference [6.37512592611305]
In deep Gaussian processes (DGPs) a sparse integration location called inducing points are selected to approximate the posterior distribution of the model. Traditional variational inference approaches to posterior approximation often lead to significant bias. We propose an alternative method called Denoising Diffusion Variational Inference (DDVI) that uses a denoising diffusion differential equation (SDE) to generate posterior samples of inducing variables.
arXiv Detail & Related papers (2024-07-24T06:39:58Z)
Closed-form Filtering for Non-linear Systems [83.91296397912218]
We propose a new class of filters based on Gaussian PSD Models, which offer several advantages in terms of density approximation and computational efficiency. We show that filtering can be efficiently performed in closed form when transitions and observations are Gaussian PSD Models. Our proposed estimator enjoys strong theoretical guarantees, with estimation error that depends on the quality of the approximation and is adaptive to the regularity of the transition probabilities.
arXiv Detail & Related papers (2024-02-15T08:51:49Z)
Projecting basis functions with tensor networks for Gaussian process regression [5.482420806459269]
We develop an approach that allows us to use an exponential amount of basis functions without the corresponding exponential computational complexity. We project the resulting weights back to the original space to make GP predictions. In an experiment with an 18-dimensional benchmark data set, we show the applicability of our method to an inverse dynamics problem.
arXiv Detail & Related papers (2023-10-31T16:59:07Z)
Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes [23.87733307119697]
We introduce Neural Operator Variational Inference (NOVI) for Deep Gaussian Processes. NOVI uses a neural generator to obtain a sampler and minimizes the Regularized Stein Discrepancy in L2 space between the generated distribution and true posterior. We demonstrate that the bias introduced by our method can be controlled by multiplying the divergence with a constant, which leads to robust error control and ensures the stability and precision of the algorithm.
arXiv Detail & Related papers (2023-09-22T06:56:35Z)
Sobolev Space Regularised Pre Density Models [51.558848491038916]
We propose a new approach to non-parametric density estimation that is based on regularizing a Sobolev norm of the density. This method is statistically consistent, and makes the inductive validation model clear and consistent.
arXiv Detail & Related papers (2023-07-25T18:47:53Z)
Score-based Diffusion Models in Function Space [137.70916238028306]
Diffusion models have recently emerged as a powerful framework for generative modeling.<n>This work introduces a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space.<n>We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution.
arXiv Detail & Related papers (2023-02-14T23:50:53Z)
Adaptive deep density approximation for fractional Fokker-Planck equations [6.066542157374599]
We present an explicit PDF model induced by a flow-based deep generative model, KRnet, which constructs a transport map from a simple distribution to the target distribution. We consider two methods to approximate the fractional Laplacian. Based on these two different ways for the approximation of the fractional Laplacian, we propose two models, MCNF and GRBFNF, to approximate stationary FPEs and time-dependent FPEs.
arXiv Detail & Related papers (2022-10-26T00:58:17Z)
Scaling Structured Inference with Randomization [64.18063627155128]
We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
arXiv Detail & Related papers (2021-12-07T11:26:41Z)
Hybrid Trilinear and Bilinear Programming for Aligning Partially Overlapping Point Sets [85.71360365315128]
In many applications, we need algorithms which can align partially overlapping point sets are invariant to the corresponding corresponding RPM algorithm. We first show that the objective is a cubic bound function. We then utilize the convex envelopes of trilinear and bilinear monomial transformations to derive its lower bound. We next develop a branch-and-bound (BnB) algorithm which only branches over the transformation variables and runs efficiently.
arXiv Detail & Related papers (2021-01-19T04:24:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.