Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting

• URL: http://arxiv.org/abs/2405.16312v2
• Date: Sun, 14 Jul 2024 14:40:20 GMT
• Title: Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting
• Authors: Jiaxi Hu, Disen Lan, Ziyu Zhou, Qingsong Wen, Yuxuan Liang,
• Abstract summary: State Space Models (SSMs) approximate continuous systems using a set of basis functions and discretize them to handle input data. This paper proposes a novel theoretical framework termed Dynamic Spectral Operator, offering more intuitive and general guidance on applying SSMs to time series data. We introduce Time-SSM, a novel SSM-based foundation model with only one-seventh of the parameters compared to Mamba.
• Score: 22.84798547604491
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State Space Models (SSMs) have emerged as a potent tool in sequence modeling tasks in recent years. These models approximate continuous systems using a set of basis functions and discretize them to handle input data, making them well-suited for modeling time series data collected at specific frequencies from continuous systems. Despite its potential, the application of SSMs in time series forecasting remains underexplored, with most existing models treating SSMs as a black box for capturing temporal or channel dependencies. To address this gap, this paper proposes a novel theoretical framework termed Dynamic Spectral Operator, offering more intuitive and general guidance on applying SSMs to time series data. Building upon our theory, we introduce Time-SSM, a novel SSM-based foundation model with only one-seventh of the parameters compared to Mamba. Various experiments validate both our theoretical framework and the superior performance of Time-SSM.

Related papers

• Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts [103.725112190618]
  This paper introduces Moirai-MoE, using a single input/output projection layer while delegating the modeling of diverse time series patterns to the sparse mixture of experts. Extensive experiments on 39 datasets demonstrate the superiority of Moirai-MoE over existing foundation models in both in-distribution and zero-shot scenarios.
  arXiv  Detail & Related papers  (2024-10-14T13:01:11Z)
• Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models [5.37935922811333]
  State Space Models (SSMs) are classical approaches for univariate time series modeling. We present Chimera that uses two input-dependent 2-D SSM heads with different discretization processes to learn long-term progression and seasonal patterns. Our experimental evaluation shows the superior performance of Chimera on extensive and diverse benchmarks.
  arXiv  Detail & Related papers  (2024-06-06T17:58:09Z)
• HOPE for a Robust Parameterization of Long-memory State Space Models [51.66430224089725]
  State-space models (SSMs) that utilize linear, time-invariant (LTI) systems are known for their effectiveness in learning long sequences. We develop a new parameterization scheme, called HOPE, for LTI systems that utilize Markov parameters within Hankel operators. Our new parameterization endows the SSM with non-decaying memory within a fixed time window, which is empirically corroborated by a sequential CIFAR-10 task with padded noise.
  arXiv  Detail & Related papers  (2024-05-22T20:20:14Z)
• State Space Models as Foundation Models: A Control Theoretic Overview [3.3222241150972356]
  In recent years, there has been a growing interest in integrating linear state-space models (SSM) in deep neural network architectures. This paper is intended as a gentle introduction to SSM-based architectures for control theorists. It provides a systematic review of the most successful SSM proposals and highlights their main features from a control theoretic perspective.
  arXiv  Detail & Related papers  (2024-03-25T16:10:47Z)
• PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the perspective of partial differential equations [49.80959046861793]
  We present PDETime, a novel LMTF model inspired by the principles of Neural PDE solvers. Our experimentation across seven diversetemporal real-world LMTF datasets reveals that PDETime adapts effectively to the intrinsic nature of the data.
  arXiv  Detail & Related papers  (2024-02-25T17:39:44Z)
• Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective [63.60312929416228]
  textbftextitAttraos incorporates chaos theory into long-term time series forecasting. We show that Attraos outperforms various LTSF methods on mainstream datasets and chaotic datasets with only one-twelfth of the parameters compared to PatchTST.
  arXiv  Detail & Related papers  (2024-02-18T05:35:01Z)
• Efficient Exploration in Continuous-time Model-based Reinforcement Learning [37.14026153342745]
  Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. We introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics.
  arXiv  Detail & Related papers  (2023-10-30T15:04:40Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• Neural Continuous-Discrete State Space Models for Irregularly-Sampled Time Series [18.885471782270375]
  NCDSSM employs auxiliary variables to disentangle recognition from dynamics, thus requiring amortized inference only for the auxiliary variables. We propose three flexible parameterizations of the latent dynamics and an efficient training objective that marginalizes the dynamic states during inference. Empirical results on multiple benchmark datasets show improved imputation and forecasting performance of NCDSSM over existing models.
  arXiv  Detail & Related papers  (2023-01-26T18:45:04Z)
• Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting [13.441945545904504]
  We propose a novel framework, in which STGNN is Enhanced by a scalable time series Pre-training model (STEP) Specifically, we design a pre-training model to efficiently learn temporal patterns from very long-term history time series. Our framework is capable of significantly enhancing downstream STGNNs, and our pre-training model aptly captures temporal patterns.
  arXiv  Detail & Related papers  (2022-06-18T04:24:36Z)
• Model-Attentive Ensemble Learning for Sequence Modeling [86.4785354333566]
  We present Model-Attentive Ensemble learning for Sequence modeling (MAES) MAES is a mixture of time-series experts which leverages an attention-based gating mechanism to specialize the experts on different sequence dynamics and adaptively weight their predictions. We demonstrate that MAES significantly out-performs popular sequence models on datasets subject to temporal shift.
  arXiv  Detail & Related papers  (2021-02-23T05:23:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.