A Deterministic Approximation to Neural SDEs
- URL: http://arxiv.org/abs/2006.08973v6
- Date: Mon, 12 Sep 2022 07:04:38 GMT
- Title: A Deterministic Approximation to Neural SDEs
- Authors: Andreas Look, Melih Kandemir, Barbara Rakitsch, Jan Peters
- Abstract summary: We show that obtaining well-calibrated uncertainty estimations from NSDEs is computationally prohibitive.
We develop a computationally affordable deterministic scheme which accurately approximates the transition kernel.
Our method also improves prediction accuracy thanks to the numerical stability of deterministic training.
- Score: 38.23826389188657
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural Stochastic Differential Equations (NSDEs) model the drift and
diffusion functions of a stochastic process as neural networks. While NSDEs are
known to make accurate predictions, their uncertainty quantification properties
have been remained unexplored so far. We report the empirical finding that
obtaining well-calibrated uncertainty estimations from NSDEs is computationally
prohibitive. As a remedy, we develop a computationally affordable deterministic
scheme which accurately approximates the transition kernel, when dynamics is
governed by a NSDE. Our method introduces a bidimensional moment matching
algorithm: vertical along the neural net layers and horizontal along the time
direction, which benefits from an original combination of effective
approximations. Our deterministic approximation of the transition kernel is
applicable to both training and prediction. We observe in multiple experiments
that the uncertainty calibration quality of our method can be matched by Monte
Carlo sampling only after introducing high computational cost. Thanks to the
numerical stability of deterministic training, our method also improves
prediction accuracy.
Related papers
- On the Hardness of Probabilistic Neurosymbolic Learning [10.180468225166441]
We study the complexity of differentiating probabilistic reasoning in neurosymbolic models.
We introduce WeightME, an unbiased gradient estimator based on model sampling.
Our experiments indicate that the existing biased approximations indeed struggle to optimize even when exact solving is still feasible.
arXiv Detail & Related papers (2024-06-06T19:56:33Z) - Noise in the reverse process improves the approximation capabilities of
diffusion models [27.65800389807353]
In Score based Generative Modeling (SGMs), the state-of-the-art in generative modeling, reverse processes are known to perform better than their deterministic counterparts.
This paper delves into the heart of this phenomenon, comparing neural ordinary differential equations (ODEs) and neural dimension equations (SDEs) as reverse processes.
We analyze the ability of neural SDEs to approximate trajectories of the Fokker-Planck equation, revealing the advantages of neurality.
arXiv Detail & Related papers (2023-12-13T02:39:10Z) - Non-Parametric Learning of Stochastic Differential Equations with Non-asymptotic Fast Rates of Convergence [65.63201894457404]
We propose a novel non-parametric learning paradigm for the identification of drift and diffusion coefficients of non-linear differential equations.
The key idea essentially consists of fitting a RKHS-based approximation of the corresponding Fokker-Planck equation to such observations.
arXiv Detail & Related papers (2023-05-24T20:43:47Z) - Monte Carlo Neural PDE Solver for Learning PDEs via Probabilistic Representation [59.45669299295436]
We propose a Monte Carlo PDE solver for training unsupervised neural solvers.
We use the PDEs' probabilistic representation, which regards macroscopic phenomena as ensembles of random particles.
Our experiments on convection-diffusion, Allen-Cahn, and Navier-Stokes equations demonstrate significant improvements in accuracy and efficiency.
arXiv Detail & Related papers (2023-02-10T08:05:19Z) - Scalable computation of prediction intervals for neural networks via
matrix sketching [79.44177623781043]
Existing algorithms for uncertainty estimation require modifying the model architecture and training procedure.
This work proposes a new algorithm that can be applied to a given trained neural network and produces approximate prediction intervals.
arXiv Detail & Related papers (2022-05-06T13:18:31Z) - NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural
Networks [151.03112356092575]
We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution.
We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
arXiv Detail & Related papers (2022-02-07T12:30:45Z) - Dense Uncertainty Estimation [62.23555922631451]
In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation.
We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
arXiv Detail & Related papers (2021-10-13T01:23:48Z) - Distributional Gradient Matching for Learning Uncertain Neural Dynamics
Models [38.17499046781131]
We propose a novel approach towards estimating uncertain neural ODEs, avoiding the numerical integration bottleneck.
Our algorithm - distributional gradient matching (DGM) - jointly trains a smoother and a dynamics model and matches their gradients via minimizing a Wasserstein loss.
Our experiments show that, compared to traditional approximate inference methods based on numerical integration, our approach is faster to train, faster at predicting previously unseen trajectories, and in the context of neural ODEs, significantly more accurate.
arXiv Detail & Related papers (2021-06-22T08:40:51Z) - Stable Neural Flows [15.318500611972441]
We introduce a provably stable variant of neural ordinary differential equations (neural ODEs) whose trajectories evolve on an energy functional parametrised by a neural network.
The learning procedure is cast as an optimal control problem, and an approximate solution is proposed based on adjoint sensivity analysis.
arXiv Detail & Related papers (2020-03-18T06:27:21Z) - Stochastic gradient descent with random learning rate [0.0]
We propose to optimize neural networks with a uniformly-distributed random learning rate.
By comparing the random learning rate protocol with cyclic and constant protocols, we suggest that the random choice is generically the best strategy in the small learning rate regime.
We provide supporting evidence through experiments on both shallow, fully-connected and deep, convolutional neural networks for image classification on the MNIST and CIFAR10 datasets.
arXiv Detail & Related papers (2020-03-15T21:36:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.