Foundation Inference Models for Stochastic Differential Equations: A Transformer-based Approach for Zero-shot Function Estimation

• URL: http://arxiv.org/abs/2502.19049v1
• Date: Wed, 26 Feb 2025 11:04:02 GMT
• Title: Foundation Inference Models for Stochastic Differential Equations: A Transformer-based Approach for Zero-shot Function Estimation
• Authors: Patrick Seifner, Kostadin Cvejoski, David Berghaus, Cesar Ojeda, Ramses J. Sanchez,
• Abstract summary: We introduce FIM-SDE (Foundation Inference Model for SDEs), a transformer-based recognition model capable of performing accurate zero-shot estimation of the drift and diffusion functions of SDEs.<n>We demonstrate that one and the same (pretrained) FIM-SDE achieves robust zero-shot function estimation across a wide range of synthetic and real-world processes.
• Score: 3.005912045854039
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Stochastic differential equations (SDEs) describe dynamical systems where deterministic flows, governed by a drift function, are superimposed with random fluctuations dictated by a diffusion function. The accurate estimation (or discovery) of these functions from data is a central problem in machine learning, with wide application across natural and social sciences alike. Yet current solutions are brittle, and typically rely on symbolic regression or Bayesian non-parametrics. In this work, we introduce FIM-SDE (Foundation Inference Model for SDEs), a transformer-based recognition model capable of performing accurate zero-shot estimation of the drift and diffusion functions of SDEs, from noisy and sparse observations on empirical processes of different dimensionalities. Leveraging concepts from amortized inference and neural operators, we train FIM-SDE in a supervised fashion, to map a large set of noisy and discretely observed SDE paths to their corresponding drift and diffusion functions. We demonstrate that one and the same (pretrained) FIM-SDE achieves robust zero-shot function estimation (i.e. without any parameter fine-tuning) across a wide range of synthetic and real-world processes, from canonical SDE systems (e.g. double-well dynamics or weakly perturbed Hopf bifurcations) to human motion recordings and oil price and wind speed fluctuations.

Related papers

• Principled model selection for stochastic dynamics [0.0]
  PASTIS is a principled method combining likelihood-estimation statistics with extreme value theory to suppress superfluous parameters.<n>It reliably identifies minimal models, even with low sampling rates or measurement error.<n>It applies to partial differential equations, and applies to ecological networks and reaction-diffusion dynamics.
  arXiv  Detail & Related papers  (2025-01-17T18:23:16Z)
• A Data-Driven Framework for Discovering Fractional Differential Equations in Complex Systems [8.206685537936078]
  This study introduces a stepwise data-driven framework for discovering fractional differential equations (FDEs) directly from data.<n>Our framework applies deep neural networks as surrogate models for denoising and reconstructing sparse and noisy observations.<n>We validate the framework across various datasets, including synthetic anomalous diffusion data and experimental data on the creep behavior of frozen soils.
  arXiv  Detail & Related papers  (2024-12-05T08:38:30Z)
• Identifying Drift, Diffusion, and Causal Structure from Temporal Snapshots [10.018568337210876]
  We present the first comprehensive approach for jointly estimating the drift and diffusion of an SDE from its temporal marginals. We show that each of these steps areAlterally optimal with respect to the Kullback-Leibler datasets.
  arXiv  Detail & Related papers  (2024-10-30T06:28:21Z)
• A Training-Free Conditional Diffusion Model for Learning Stochastic Dynamical Systems [10.820654486318336]
  This study introduces a training-free conditional diffusion model for learning unknown differential equations (SDEs) using data. The proposed approach addresses key challenges in computational efficiency and accuracy for modeling SDEs. The learned models exhibit significant improvements in predicting both short-term and long-term behaviors of unknown systems.
  arXiv  Detail & Related papers  (2024-10-04T03:07:36Z)
• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• Diffusion models as probabilistic neural operators for recovering unobserved states of dynamical systems [49.2319247825857]
  We show that diffusion-based generative models exhibit many properties favourable for neural operators.<n>We propose to train a single model adaptable to multiple tasks, by alternating between the tasks during training.
  arXiv  Detail & Related papers  (2024-05-11T21:23:55Z)
• Neural McKean-Vlasov Processes: Distributional Dependence in Diffusion Processes [24.24785205800212]
  McKean-Vlasov differential equations (MV-SDEs) provide a mathematical description of the behavior of an infinite number of interacting particles. We study the influence of explicitly including distributional information in the parameterization of the SDE.
  arXiv  Detail & Related papers  (2024-04-15T01:28:16Z)
• Gaussian Mixture Solvers for Diffusion Models [84.83349474361204]
  We introduce a novel class of SDE-based solvers called GMS for diffusion models. Our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis.
  arXiv  Detail & Related papers  (2023-11-02T02:05:38Z)
• Variational Inference for SDEs Driven by Fractional Noise [16.434973057669676]
  We present a novel variational framework for performing inference in (neural) differential equations (SDEs) driven by Markov-approximate fractional Brownian motion (fBM) We propose the use of neural networks to learn the drift, diffusion and control terms within our variational posterior leading to the variational training of neural-SDEs.
  arXiv  Detail & Related papers  (2023-10-19T17:59:21Z)
• A Geometric Perspective on Diffusion Models [57.27857591493788]
  We inspect the ODE-based sampling of a popular variance-exploding SDE. We establish a theoretical relationship between the optimal ODE-based sampling and the classic mean-shift (mode-seeking) algorithm.
  arXiv  Detail & Related papers  (2023-05-31T15:33:16Z)
• Score-based Diffusion Models in Function Space [137.70916238028306]
  Diffusion models have recently emerged as a powerful framework for generative modeling.<n>This work introduces a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) for training diffusion models in function space.<n>We show that the corresponding discretized algorithm generates accurate samples at a fixed cost independent of the data resolution.
  arXiv  Detail & Related papers  (2023-02-14T23:50:53Z)
• Stochastic Normalizing Flows [52.92110730286403]
  We introduce normalizing flows for maximum likelihood estimation and variational inference (VI) using differential equations (SDEs) Using the theory of rough paths, the underlying Brownian motion is treated as a latent variable and approximated, enabling efficient training of neural SDEs. These SDEs can be used for constructing efficient chains to sample from the underlying distribution of a given dataset.
  arXiv  Detail & Related papers  (2020-02-21T20:47:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.