Sparse Transformer with Local and Seasonal Adaptation for Multivariate Time Series Forecasting

• URL: http://arxiv.org/abs/2312.06874v2
• Date: Mon, 15 Jul 2024 20:59:42 GMT
• Title: Sparse Transformer with Local and Seasonal Adaptation for Multivariate Time Series Forecasting
• Authors: Yifan Zhang, Rui Wu, Sergiu M. Dascalu, Frederick C. Harris Jr,
• Abstract summary: We propose a Dozer Attention mechanism consisting of three sparse components. These components are designed to capture essential attributes of MTS data, including locality, seasonality, and global temporal dependencies. We present the Dozerformer Framework, incorporating the Dozer Attention mechanism for the MTS forecasting task.
• Score: 8.000134983886742
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Transformers have achieved remarkable performance in multivariate time series(MTS) forecasting due to their capability to capture long-term dependencies. However, the canonical attention mechanism has two key limitations: (1) its quadratic time complexity limits the sequence length, and (2) it generates future values from the entire historical sequence. To address this, we propose a Dozer Attention mechanism consisting of three sparse components: (1) Local, each query exclusively attends to keys within a localized window of neighboring time steps. (2) Stride, enables each query to attend to keys at predefined intervals. (3) Vary, allows queries to selectively attend to keys from a subset of the historical sequence. Notably, the size of this subset dynamically expands as forecasting horizons extend. Those three components are designed to capture essential attributes of MTS data, including locality, seasonality, and global temporal dependencies. Additionally, we present the Dozerformer Framework, incorporating the Dozer Attention mechanism for the MTS forecasting task. We evaluated the proposed Dozerformer framework with recent state-of-the-art methods on nine benchmark datasets and confirmed its superior performance. The experimental results indicate that excluding a subset of historical time steps from the time series forecasting process does not compromise accuracy while significantly improving efficiency. Code is available at https://github.com/GRYGY1215/Dozerformer.

Related papers

• ViTs: Teaching Machines to See Time Series Anomalies Like Human Experts [21.498848897981173]
  "Train once, infer across scenarios" remains a fundamental challenge for time series anomaly detection models.<n>We propose ViTs, a Vision-Language Model (VLM)-based framework that converts time series curves into visual representations.
  arXiv  Detail & Related papers  (2025-10-06T11:24:53Z)
• TimeExpert: Boosting Long Time Series Forecasting with Temporal Mix of Experts [11.53964887034519]
  We propose the Temporal Mix of Experts (TMOE), a novel attention-level mechanism that reimagines key-value (K-V) pairs as local experts.<n>TMOE performs adaptive expert selection for each query via localized filtering of irrelevant timestamps.<n>We then replace the vanilla attention mechanism in popular time-series Transformer frameworks (i.e., PatchTST and Timer) with TMOE, without extra structural modifications.
  arXiv  Detail & Related papers  (2025-09-27T06:22:09Z)
• T3Time: Tri-Modal Time Series Forecasting via Adaptive Multi-Head Alignment and Residual Fusion [0.4915744683251151]
  T3Time is a novel trimodal framework consisting of time, spectral, and prompt branches.<n>It learns prioritization between temporal and spectral features based on the prediction horizon.<n>Our model consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-06T09:31:44Z)
• PSformer: Parameter-efficient Transformer with Segment Attention for Time Series Forecasting [21.033660755921737]
  Time forecasting remains a critical challenge across various domains, often complicated by high-dimensional data and long-term dependencies. This paper presents a novel transformer architecture for time series forecasting, incorporating two key innovations: parameter sharing (PS) and Spatial-Temporal Attention (SegAtt)
  arXiv  Detail & Related papers  (2024-11-03T03:04:00Z)
• Timer-XL: Long-Context Transformers for Unified Time Series Forecasting [67.83502953961505]
  We present Timer-XL, a generative Transformer for unified time series forecasting. Timer-XL achieves state-of-the-art performance across challenging forecasting benchmarks through a unified approach.
  arXiv  Detail & Related papers  (2024-10-07T07:27:39Z)
• TSCMamba: Mamba Meets Multi-View Learning for Time Series Classification [13.110156202816112]
  We propose a novel multi-view approach to capture patterns with properties like shift equivariance. Our method integrates diverse features, including spectral, temporal, local, and global features, to obtain rich, complementary contexts for TSC. Our approach achieves average accuracy improvements of 4.01-6.45% and 7.93% respectively, over leading TSC models.
  arXiv  Detail & Related papers  (2024-06-06T18:05:10Z)
• FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification [53.55504611255664]
  FormerTime is a hierarchical representation model for improving the classification capacity for the multivariate time series classification task. It exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism.
  arXiv  Detail & Related papers  (2023-02-20T07:46:14Z)
• Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement [51.55157852647306]
  Time series forecasting has been a widely explored task of great importance in many applications. It is common that real-world time series data are recorded in a short time period, which results in a big gap between the deep model and the limited and noisy time series. We propose to address the time series forecasting problem with generative modeling and propose a bidirectional variational auto-encoder equipped with diffusion, denoise, and disentanglement.
  arXiv  Detail & Related papers  (2023-01-08T12:20:46Z)
• Respecting Time Series Properties Makes Deep Time Series Forecasting Perfect [3.830797055092574]
  How to handle time features shall be the core question of any time series forecasting model. In this paper, we rigorously analyze three prevalent but deficient/unfounded deep time series forecasting mechanisms. We propose a novel time series forecasting network, i.e. RTNet, on the basis of aforementioned analysis.
  arXiv  Detail & Related papers  (2022-07-22T08:34:31Z)
• Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Version [50.43914511877446]
  We propose a triangular, variable-specific attention to ensure high efficiency and accuracy. We show that Triformer outperforms state-of-the-art methods w.r.t. both accuracy and efficiency.
  arXiv  Detail & Related papers  (2022-04-28T20:41:49Z)
• Cluster-and-Conquer: A Framework For Time-Series Forecasting [94.63501563413725]
  We propose a three-stage framework for forecasting high-dimensional time-series data. Our framework is highly general, allowing for any time-series forecasting and clustering method to be used in each step. When instantiated with simple linear autoregressive models, we are able to achieve state-of-the-art results on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-10-26T20:41:19Z)
• Complex Event Forecasting with Prediction Suffix Trees: Extended Technical Report [70.7321040534471]
  Complex Event Recognition (CER) systems have become popular in the past two decades due to their ability to "instantly" detect patterns on real-time streams of events. There is a lack of methods for forecasting when a pattern might occur before such an occurrence is actually detected by a CER engine. We present a formal framework that attempts to address the issue of Complex Event Forecasting.
  arXiv  Detail & Related papers  (2021-09-01T09:52:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.