Amortized Reparametrization: Efficient and Scalable Variational Inference for Latent SDEs

• URL: http://arxiv.org/abs/2312.10550v1
• Date: Sat, 16 Dec 2023 22:27:36 GMT
• Title: Amortized Reparametrization: Efficient and Scalable Variational Inference for Latent SDEs
• Authors: Kevin Course, Prasanth B. Nair
• Abstract summary: We consider the problem of inferring latent differential equations with a time and memory cost that scales independently with the amount of data, the total length of the time series, and the stiffness of the approximate differential equations. This is in stark contrast to typical methods for inferring latent differential equations which, despite their constant memory cost, have a time complexity that is heavily dependent on the stiffness of the approximate differential equation.
• Score: 3.2634122554914002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider the problem of inferring latent stochastic differential equations (SDEs) with a time and memory cost that scales independently with the amount of data, the total length of the time series, and the stiffness of the approximate differential equations. This is in stark contrast to typical methods for inferring latent differential equations which, despite their constant memory cost, have a time complexity that is heavily dependent on the stiffness of the approximate differential equation. We achieve this computational advancement by removing the need to solve differential equations when approximating gradients using a novel amortization strategy coupled with a recently derived reparametrization of expectations under linear SDEs. We show that, in practice, this allows us to achieve similar performance to methods based on adjoint sensitivities with more than an order of magnitude fewer evaluations of the model in training.

Related papers

• From continuous-time formulations to discretization schemes: tensor trains and robust regression for BSDEs and parabolic PDEs [3.785123406103385]
  We argue that tensor trains provide an appealing framework for parabolic PDEs. We develop iterative schemes, which differ in terms of computational efficiency and robustness. We demonstrate both theoretically and numerically that our methods can achieve a favorable trade-off between accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2023-07-28T11:44:06Z)
• A Geometric Perspective on Diffusion Models [57.27857591493788]
  We inspect the ODE-based sampling of a popular variance-exploding SDE. We establish a theoretical relationship between the optimal ODE-based sampling and the classic mean-shift (mode-seeking) algorithm.
  arXiv  Detail & Related papers  (2023-05-31T15:33:16Z)
• Non-Parametric Learning of Stochastic Differential Equations with Non-asymptotic Fast Rates of Convergence [65.63201894457404]
  We propose a novel non-parametric learning paradigm for the identification of drift and diffusion coefficients of non-linear differential equations. The key idea essentially consists of fitting a RKHS-based approximation of the corresponding Fokker-Planck equation to such observations.
  arXiv  Detail & Related papers  (2023-05-24T20:43:47Z)
• Self-Consistent Velocity Matching of Probability Flows [22.2542921090435]
  We present a discretization-free scalable framework for solving a class of partial differential equations (PDEs) The main observation is that the time-varying velocity field of the PDE solution needs to be self-consistent. We use an iterative formulation with a biased gradient estimator that bypasses significant computational obstacles with strong empirical performance.
  arXiv  Detail & Related papers  (2023-01-31T16:17:18Z)
• Symbolic Recovery of Differential Equations: The Identifiability Problem [52.158782751264205]
  Symbolic recovery of differential equations is the ambitious attempt at automating the derivation of governing equations. We provide both necessary and sufficient conditions for a function to uniquely determine the corresponding differential equation. We then use our results to devise numerical algorithms aiming to determine whether a function solves a differential equation uniquely.
  arXiv  Detail & Related papers  (2022-10-15T17:32:49Z)
• Robust SDE-Based Variational Formulations for Solving Linear PDEs via Deep Learning [6.1678491628787455]
  Combination of Monte Carlo methods and deep learning has led to efficient algorithms for solving partial differential equations (PDEs) in high dimensions. Related learning problems are often stated as variational formulations based on associated differential equations (SDEs) It is therefore crucial to rely on adequate gradient estimators that exhibit low variance in order to reach convergence accurately and swiftly.
  arXiv  Detail & Related papers  (2022-06-21T17:59:39Z)
• D-CIPHER: Discovery of Closed-form Partial Differential Equations [80.46395274587098]
  We propose D-CIPHER, which is robust to measurement artifacts and can uncover a new and very general class of differential equations. We further design a novel optimization procedure, CoLLie, to help D-CIPHER search through this class efficiently.
  arXiv  Detail & Related papers  (2022-06-21T17:59:20Z)
• Robust and Scalable SDE Learning: A Functional Perspective [5.642000444047032]
  We propose an importance-sampling for probabilities of observations of SDE estimators for the purposes of learning. The proposed method produces lower-variance estimates compared to algorithms based on SDE. This facilitates the effective use of large-scale parallel hardware for massive decreases in time.
  arXiv  Detail & Related papers  (2021-10-11T11:36:50Z)
• The Connection between Discrete- and Continuous-Time Descriptions of Gaussian Continuous Processes [60.35125735474386]
  We show that discretizations yielding consistent estimators have the property of invariance under coarse-graining' This result explains why combining differencing schemes for derivatives reconstruction and local-in-time inference approaches does not work for time series analysis of second or higher order differential equations.
  arXiv  Detail & Related papers  (2021-01-16T17:11:02Z)
• On Learning Rates and Schr\"odinger Operators [105.32118775014015]
  We present a general theoretical analysis of the effect of the learning rate. We find that the learning rate tends to zero for a broad non- neural class functions.
  arXiv  Detail & Related papers  (2020-04-15T09:52:37Z)
• Scalable Gradients for Stochastic Differential Equations [40.70998833051251]
  adjoint sensitivity method scalably computes gradients of ordinary differential equations. We generalize this method to differential equations, allowing time-efficient and constant-memory computation. We use our method to fit neural dynamics defined by networks, achieving competitive performance on a 50-dimensional motion capture dataset.
  arXiv  Detail & Related papers  (2020-01-05T23:05:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.