Delayformer: spatiotemporal transformation for predicting high-dimensional dynamics

• URL: http://arxiv.org/abs/2506.11528v1
• Date: Fri, 13 Jun 2025 07:30:12 GMT
• Title: Delayformer: spatiotemporal transformation for predicting high-dimensional dynamics
• Authors: Zijian Wang, Peng Tao, Luonan Chen,
• Abstract summary: Delayformer is a framework for simultaneously predicting dynamics of all variables.<n>From dynamical systems viewpoint, Delayformer predicts system states rather than individual variables.<n> Delayformer outperforms current state-of-the-art methods in forecasting tasks on both synthetic and real-world datasets.
• Score: 4.984442247236504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting time-series is of great importance in various scientific and engineering fields. However, in the context of limited and noisy data, accurately predicting dynamics of all variables in a high-dimensional system is a challenging task due to their nonlinearity and also complex interactions. Current methods including deep learning approaches often perform poorly for real-world systems under such circumstances. This study introduces the Delayformer framework for simultaneously predicting dynamics of all variables, by developing a novel multivariate spatiotemporal information (mvSTI) transformation that makes each observed variable into a delay-embedded state (vector) and further cross-learns those states from different variables. From dynamical systems viewpoint, Delayformer predicts system states rather than individual variables, thus theoretically and computationally overcoming such nonlinearity and cross-interaction problems. Specifically, it first utilizes a single shared Visual Transformer (ViT) encoder to cross-represent dynamical states from observed variables in a delay embedded form and then employs distinct linear decoders for predicting next states, i.e. equivalently predicting all original variables parallelly. By leveraging the theoretical foundations of delay embedding theory and the representational capabilities of Transformers, Delayformer outperforms current state-of-the-art methods in forecasting tasks on both synthetic and real-world datasets. Furthermore, the potential of Delayformer as a foundational time-series model is demonstrated through cross-domain forecasting tasks, highlighting its broad applicability across various scenarios.

Related papers

• Spatio-temporal Fourier Transformer (StFT) for Long-term Dynamics Prediction [13.502705948250359]
  Simulating the long-term dynamics of multi-scale and multi-physics systems poses a significant challenge in science and engineering.<n>Neural operators have emerged as promising models for predicting such dynamics due to their flexibility and computational efficiency.<n>We propose a generative residual correction mechanism to estimate and mitigate predictive uncertainties, enhancing the accuracy and reliability of long-term forecasts.
  arXiv  Detail & Related papers  (2025-03-14T22:04:03Z)
• Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models [71.63194926457119]
  We introduce Dynamical Diffusion (DyDiff), a theoretically sound framework that incorporates temporally aware forward and reverse processes.<n>Experiments across scientifictemporal forecasting, video prediction, and time series forecasting demonstrate that Dynamical Diffusion consistently improves performance in temporal predictive tasks.
  arXiv  Detail & Related papers  (2025-03-02T16:10:32Z)
• EDformer: Embedded Decomposition Transformer for Interpretable Multivariate Time Series Predictions [4.075971633195745]
  This paper introduces an embedded transformer, 'EDformer', for time series forecasting tasks.<n>Without altering the fundamental elements, we reuse the Transformer architecture and consider the capable functions of its constituent parts.<n>The model obtains state-of-the-art predicting results in terms of accuracy and efficiency on complex real-world time series datasets.
  arXiv  Detail & Related papers  (2024-12-16T11:13:57Z)
• Timer-XL: Long-Context Transformers for Unified Time Series Forecasting [67.83502953961505]
  We present Timer-XL, a causal Transformer for unified time series forecasting.<n>Based on large-scale pre-training, Timer-XL achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-10-07T07:27:39Z)
• TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model [11.281386703572842]
  TimeDiT is a diffusion transformer model that combines temporal dependency learning with probabilistic sampling.<n>TimeDiT employs a unified masking mechanism to harmonize the training and inference process across diverse tasks.<n>Our systematic evaluation demonstrates TimeDiT's effectiveness both in fundamental tasks, i.e., forecasting and imputation, through zero-shot/fine-tuning.
  arXiv  Detail & Related papers  (2024-09-03T22:31:57Z)
• TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables [75.83318701911274]
  TimeXer ingests external information to enhance the forecasting of endogenous variables. TimeXer achieves consistent state-of-the-art performance on twelve real-world forecasting benchmarks.
  arXiv  Detail & Related papers  (2024-02-29T11:54:35Z)
• Learning Latent Dynamics via Invariant Decomposition and (Spatio-)Temporal Transformers [0.6767885381740952]
  We propose a method for learning dynamical systems from high-dimensional empirical data. We focus on the setting in which data are available from multiple different instances of a system. We study behaviour through simple theoretical analyses and extensive experiments on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-06-21T07:52:07Z)
• Modulated Neural ODEs [32.2290908839375]
  We introduce Modulated Neural ODEs (MoNODEs), a novel framework that sets apart dynamics states from underlying static factors of variation. We test MoNODE on oscillating systems, videos and human walking trajectories, where each trajectory has trajectory-specific modulation.
  arXiv  Detail & Related papers  (2023-02-26T08:26:01Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Multivariate Time Series Forecasting with Dynamic Graph Neural ODEs [65.18780403244178]
  We propose a continuous model to forecast Multivariate Time series with dynamic Graph neural Ordinary Differential Equations (MTGODE) Specifically, we first abstract multivariate time series into dynamic graphs with time-evolving node features and unknown graph structures. Then, we design and solve a neural ODE to complement missing graph topologies and unify both spatial and temporal message passing.
  arXiv  Detail & Related papers  (2022-02-17T02:17:31Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)
• Temporal Tensor Transformation Network for Multivariate Time Series Prediction [1.2354076490479515]
  We present a novel deep learning architecture, known as Temporal Transformation Network, which transforms the original time series into a higher order. This yields a new representation of the original multivariate time series, which enables the convolution kernel to extract complex and non-linear features as well as variable interactional signals from a relatively large temporal region. Experimental results show that Temporal Transformation Network outperforms several state-of-the-art methods on window-based predictions across various tasks.
  arXiv  Detail & Related papers  (2020-01-04T07:28:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.