Time-Delayed Transformers for Data-Driven Modeling of Low-Dimensional Dynamics

• URL: http://arxiv.org/abs/2602.08478v1
• Date: Mon, 09 Feb 2026 10:22:43 GMT
• Title: Time-Delayed Transformers for Data-Driven Modeling of Low-Dimensional Dynamics
• Authors: Albert Alcalde, Markus Widhalm, Emre Yılmaz,
• Abstract summary: We present a time-delayed transformer (TD-TF) for data-driven modeling of unsteady-temporal dynamics.<n>The architecture is deliberately minimal, consisting of one self-attention layer with a single query per generalization and one feedforward layer.<n> Numerical experiments demonstrate that TD-TF matches the performance of strong linear baselines on near-linear systems, while significantly outperforming them in nonlinear and chaotic regimes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose the time-delayed transformer (TD-TF), a simplified transformer architecture for data-driven modeling of unsteady spatio-temporal dynamics. TD-TF bridges linear operator-based methods and deep sequence models by showing that a single-layer, single-head transformer can be interpreted as a nonlinear generalization of time-delayed dynamic mode decomposition (TD-DMD). The architecture is deliberately minimal, consisting of one self-attention layer with a single query per prediction and one feedforward layer, resulting in linear computational complexity in sequence length and a small parameter count. Numerical experiments demonstrate that TD-TF matches the performance of strong linear baselines on near-linear systems, while significantly outperforming them in nonlinear and chaotic regimes, where it accurately captures long-term dynamics. Validation studies on synthetic signals, unsteady aerodynamics, the Lorenz '63 system, and a reaction-diffusion model show that TD-TF preserves the interpretability and efficiency of linear models while providing substantially enhanced expressive power for complex dynamics.

Related papers

• PRISM: Parallel Residual Iterative Sequence Model [52.26239951489612]
  We propose PRISM (Parallel Residual Iterative Sequence Model) to resolve this tension.<n>PRISM introduces a solver-inspired inductive bias that captures key structural properties of multi-step refinement in a parallelizable form.<n>We prove that this formulation achieves Rank-$L$ accumulation, structurally expanding the update manifold beyond the single-step Rank-$1$ bottleneck.
  arXiv  Detail & Related papers  (2026-02-11T12:39:41Z)
• DiTS: Multimodal Diffusion Transformers Are Time Series Forecasters [50.43534351968113]
  Existing generative time series models do not address the multi-dimensional properties of time series data well.<n>Inspired by Multimodal Diffusion Transformers that integrate textual guidance into video generation, we propose Diffusion Transformers for Time Series (DiTS)
  arXiv  Detail & Related papers  (2026-02-06T10:48:13Z)
• ACFormer: Mitigating Non-linearity with Auto Convolutional Encoder for Time Series Forecasting [6.27761817493579]
  Time series forecasting (TSF) faces challenges in modeling complex intra-channel temporal dependencies and inter-channel correlations.<n>We propose ACFormer, an architecture designed to reconcile the efficiency of linear projections with the non-linear feature-extraction power of convolutions.
  arXiv  Detail & Related papers  (2026-01-28T13:47:54Z)
• Tensor Network Framework for Forecasting Nonlinear and Chaotic Dynamics [1.790605517028706]
  We present a tensor network model (TNM) for forecasting nonlinear and chaotic dynamics.<n>We show that the TNM accurately reconstructs short-term trajectories and faithfully captures the attractor geometry.
  arXiv  Detail & Related papers  (2025-11-12T11:49:38Z)
• Hybrid machine learning models based on physical patterns to accelerate CFD simulations: a short guide on autoregressive models [3.780691701083858]
  This study presents an innovative integration of High-Order Singular Value Decomposition with Long Short-Term Memory (LSTM) architectures to address the complexities of reduced-order modeling (ROM) in fluid dynamics.<n>The methodology is tested across numerical and experimental data sets, including two- and three-dimensional (2D and 3D) cylinder wake flows, spanning both laminar and turbulent regimes.<n>The results demonstrate that HOSVD outperforms SVD in all tested scenarios, as evidenced by using different error metrics.
  arXiv  Detail & Related papers  (2025-04-09T10:56:03Z)
• AverageTime: Enhance Long-Term Time Series Forecasting with Simple Averaging [6.125620036017928]
  Long-term time series forecasting focuses on leveraging historical data to predict future trends.<n>The core challenge lies in effectively modeling dependencies both within sequences and channels.<n>Our research proposes a new approach for capturing sequence and channel dependencies: AverageTime.
  arXiv  Detail & Related papers  (2024-12-30T05:56:25Z)
• Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective [63.60312929416228]
  textbftextitAttraos incorporates chaos theory into long-term time series forecasting. We show that Attraos outperforms various LTSF methods on mainstream datasets and chaotic datasets with only one-twelfth of the parameters compared to PatchTST.
  arXiv  Detail & Related papers  (2024-02-18T05:35:01Z)
• Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution [57.71199089609161]
  Long-term time-series forecasting (LTTF) has become a pressing demand in many applications, such as wind power supply planning. Transformer models have been adopted to deliver high prediction capacity because of the high computational self-attention mechanism. We propose an efficient Transformerbased model, named Conformer, which differentiates itself from existing methods for LTTF in three aspects.
  arXiv  Detail & Related papers  (2023-01-05T13:59:29Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)
• Tensorized Transformer for Dynamical Systems Modeling [0.0]
  We establish a parallel between the dynamical systems modeling and language modeling tasks. We propose a transformer-based model that incorporates geometrical properties of the data. We provide an iterative training algorithm allowing the fine-grid approximation of the conditional probabilities of high-dimensional dynamical systems.
  arXiv  Detail & Related papers  (2020-06-05T13:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.