Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models

• URL: http://arxiv.org/abs/2412.08595v1
• Date: Wed, 11 Dec 2024 18:13:55 GMT
• Title: Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models
• Authors: Jaesung R. Park, Jaewook J. Suh, Ernest K. Ryu,
• Abstract summary: In deep learning, the recently introduced state space models utilize HiPPO memory units to approximate continuous-time trajectories of input functions.<n>We establish that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions.
• Score: 9.933900714070033
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unexplored. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions, and by establishing convergence of the associated numerical discretization schemes for Riemann-integrable input functions.

Related papers

• Probability-Flow ODE in Infinite-Dimensional Function Spaces [26.795793417392037]
  We derive, for the first time, an analog of the probability-flow ODE (PF-ODE) in infinite-dimensional function spaces. We reduce the number of function evaluations while maintaining sample quality in function generation tasks, including applications to PDEs.
  arXiv  Detail & Related papers  (2025-03-13T10:01:00Z)
• Advancing Generalization in PINNs through Latent-Space Representations [71.86401914779019]
  Physics-informed neural networks (PINNs) have made significant strides in modeling dynamical systems governed by partial differential equations (PDEs) We propose PIDO, a novel physics-informed neural PDE solver designed to generalize effectively across diverse PDE configurations. We validate PIDO on a range of benchmarks, including 1D combined equations and 2D Navier-Stokes equations.
  arXiv  Detail & Related papers  (2024-11-28T13:16:20Z)
• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• Constrained Synthesis with Projected Diffusion Models [47.56192362295252]
  This paper introduces an approach to generative diffusion processes the ability to satisfy and certify compliance with constraints and physical principles. The proposed method recast the traditional process of generative diffusion as a constrained distribution problem to ensure adherence to constraints.
  arXiv  Detail & Related papers  (2024-02-05T22:18:16Z)
• Solving High-Dimensional PDEs with Latent Spectral Models [74.1011309005488]
  We present Latent Spectral Models (LSM) toward an efficient and precise solver for high-dimensional PDEs. Inspired by classical spectral methods in numerical analysis, we design a neural spectral block to solve PDEs in the latent space. LSM achieves consistent state-of-the-art and yields a relative gain of 11.5% averaged on seven benchmarks.
  arXiv  Detail & Related papers  (2023-01-30T04:58:40Z)
• Discovering ordinary differential equations that govern time-series [65.07437364102931]
  We propose a transformer-based sequence-to-sequence model that recovers scalar autonomous ordinary differential equations (ODEs) in symbolic form from time-series data of a single observed solution of the ODE. Our method is efficiently scalable: after one-time pretraining on a large set of ODEs, we can infer the governing laws of a new observed solution in a few forward passes of the model.
  arXiv  Detail & Related papers  (2022-11-05T07:07:58Z)
• FaDIn: Fast Discretized Inference for Hawkes Processes with General Parametric Kernels [82.53569355337586]
  This work offers an efficient solution to temporal point processes inference using general parametric kernels with finite support. The method's effectiveness is evaluated by modeling the occurrence of stimuli-induced patterns from brain signals recorded with magnetoencephalography (MEG) Results show that the proposed approach leads to an improved estimation of pattern latency than the state-of-the-art.
  arXiv  Detail & Related papers  (2022-10-10T12:35:02Z)
• Constraining Gaussian Processes to Systems of Linear Ordinary Differential Equations [5.33024001730262]
  LODE-GPs follow a system of linear homogeneous ODEs with constant coefficients. We show the effectiveness of LODE-GPs in a number of experiments.
  arXiv  Detail & Related papers  (2022-08-26T09:16:53Z)
• Rational Approximations of Quasi-Periodic Problems via Projected Green's Functions [0.0]
  We introduce the projected Green's function technique to study quasi-periodic systems. The technique is flexible and can be used to extract both analytic and numerical results.
  arXiv  Detail & Related papers  (2021-09-28T18:00:00Z)
• Neural SDEs as Infinite-Dimensional GANs [18.07683058213448]
  We show that the current classical approach to fitting SDEs may be approached as a special case of (Wasserstein) GANs. We obtain Neural SDEs as (in modern machine learning parlance) continuous-time generative time series models.
  arXiv  Detail & Related papers  (2021-02-06T19:59:15Z)
• Exponentially Weighted l_2 Regularization Strategy in Constructing Reinforced Second-order Fuzzy Rule-based Model [72.57056258027336]
  In the conventional Takagi-Sugeno-Kang (TSK)-type fuzzy models, constant or linear functions are usually utilized as the consequent parts of the fuzzy rules. We introduce an exponential weight approach inspired by the weight function theory encountered in harmonic analysis.
  arXiv  Detail & Related papers  (2020-07-02T15:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.