Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling

• URL: http://arxiv.org/abs/2501.18871v1
• Date: Fri, 31 Jan 2025 03:47:22 GMT
• Title: Neural SDEs as a Unified Approach to Continuous-Domain Sequence Modeling
• Authors: Macheng Shen, Chen Cheng,
• Abstract summary: We propose a novel and intuitive approach to continuous sequence modeling. Our method interprets time-series data as textitdiscrete samples from an underlying continuous dynamical system. We derive a maximum principled objective and a textitsimulation-free scheme for efficient training of our Neural SDE model.
• Score: 3.8980564330208662
• License:
• Abstract: Inspired by the ubiquitous use of differential equations to model continuous dynamics across diverse scientific and engineering domains, we propose a novel and intuitive approach to continuous sequence modeling. Our method interprets time-series data as \textit{discrete samples from an underlying continuous dynamical system}, and models its time evolution using Neural Stochastic Differential Equation (Neural SDE), where both the flow (drift) and diffusion terms are parameterized by neural networks. We derive a principled maximum likelihood objective and a \textit{simulation-free} scheme for efficient training of our Neural SDE model. We demonstrate the versatility of our approach through experiments on sequence modeling tasks across both embodied and generative AI. Notably, to the best of our knowledge, this is the first work to show that SDE-based continuous-time modeling also excels in such complex scenarios, and we hope that our work opens up new avenues for research of SDE models in high-dimensional and temporally intricate domains.

Related papers

• SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equations [2.1779479916071067]
  We propose SDE Matching, a new simulation-free method for training Latent SDEs. Our results demonstrate that SDE Matching achieves performance comparable to adjoint sensitivity methods.
  arXiv  Detail & Related papers  (2025-02-04T16:47:49Z)
• Trajectory Flow Matching with Applications to Clinical Time Series Modeling [77.58277281319253]
  Trajectory Flow Matching (TFM) trains a Neural SDE in a simulation-free manner, bypassing backpropagation through the dynamics. We demonstrate improved performance on three clinical time series datasets in terms of absolute performance and uncertainty prediction.
  arXiv  Detail & Related papers  (2024-10-28T15:54:50Z)
• Latent Space Energy-based Neural ODEs [73.01344439786524]
  This paper introduces novel deep dynamical models designed to represent continuous-time sequences. We train the model using maximum likelihood estimation with Markov chain Monte Carlo. Experimental results on oscillating systems, videos and real-world state sequences (MuJoCo) demonstrate that our model with the learnable energy-based prior outperforms existing counterparts.
  arXiv  Detail & Related papers  (2024-09-05T18:14:22Z)
• Equivariant Graph Neural Operator for Modeling 3D Dynamics [148.98826858078556]
  We propose Equivariant Graph Neural Operator (EGNO) to directly models dynamics as trajectories instead of just next-step prediction. EGNO explicitly learns the temporal evolution of 3D dynamics where we formulate the dynamics as a function over time and learn neural operators to approximate it. Comprehensive experiments in multiple domains, including particle simulations, human motion capture, and molecular dynamics, demonstrate the significantly superior performance of EGNO against existing methods.
  arXiv  Detail & Related papers  (2024-01-19T21:50:32Z)
• Learning Space-Time Continuous Neural PDEs from Partially Observed States [13.01244901400942]
  We introduce a grid-independent model learning partial differential equations (PDEs) from noisy and partial observations on irregular grids. We propose a space-time continuous latent neural PDE model with an efficient probabilistic framework and a novel design encoder for improved data efficiency and grid independence.
  arXiv  Detail & Related papers  (2023-07-09T06:53:59Z)
• Neural Ideal Large Eddy Simulation: Modeling Turbulence with Neural Stochastic Differential Equations [22.707574194338132]
  We introduce a data-driven learning framework that assimilates two powerful ideas: ideal eddy simulation (LES) from turbulence closure modeling and neural differential equations (SDE) for large modeling. We show the effectiveness of our approach on a challenging chaotic dynamical system: Kolmogorov flow at a Reynolds number of 20,000.
  arXiv  Detail & Related papers  (2023-06-01T22:16:28Z)
• Neural Delay Differential Equations: System Reconstruction and Image Classification [14.59919398960571]
  We propose a new class of continuous-depth neural networks with delay, named Neural Delay Differential Equations (NDDEs) Compared to NODEs, NDDEs have a stronger capacity of nonlinear representations. We achieve lower loss and higher accuracy not only for the data produced synthetically but also for the CIFAR10, a well-known image dataset.
  arXiv  Detail & Related papers  (2023-04-11T16:09:28Z)
• Learning PDE Solution Operator for Continuous Modeling of Time-Series [1.39661494747879]
  This work presents a partial differential equation (PDE) based framework which improves the dynamics modeling capability. We propose a neural operator that can handle time continuously without requiring iterative operations or specific grids of temporal discretization. Our framework opens up a new way for a continuous representation of neural networks that can be readily adopted for real-world applications.
  arXiv  Detail & Related papers  (2023-02-02T03:47:52Z)
• Mixed Effects Neural ODE: A Variational Approximation for Analyzing the Dynamics of Panel Data [50.23363975709122]
  We propose a probabilistic model called ME-NODE to incorporate (fixed + random) mixed effects for analyzing panel data. We show that our model can be derived using smooth approximations of SDEs provided by the Wong-Zakai theorem. We then derive Evidence Based Lower Bounds for ME-NODE, and develop (efficient) training algorithms.
  arXiv  Detail & Related papers  (2022-02-18T22:41:51Z)
• Closed-form Continuous-Depth Models [99.40335716948101]
  Continuous-depth neural models rely on advanced numerical differential equation solvers. We present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster.
  arXiv  Detail & Related papers  (2021-06-25T22:08:51Z)
• Sparse Flows: Pruning Continuous-depth Models [107.98191032466544]
  We show that pruning improves generalization for neural ODEs in generative modeling. We also show that pruning finds minimal and efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy.
  arXiv  Detail & Related papers  (2021-06-24T01:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.