WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting

• URL: http://arxiv.org/abs/2410.22649v2
• Date: Thu, 21 Nov 2024 03:34:44 GMT
• Title: WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting
• Authors: Aobo Liang, Yan Sun, Nadra Guizani,
• Abstract summary: We propose a wavelet learning framework to model complex temporal dependencies of the time series data. The wavelet domain integrates both time and frequency information, allowing for the analysis of local characteristics of signals at different scales. We propose a novel attention mechanism: Rotary Route Attention (RoRA)
• Score: 4.680374146155483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, Transformer-based models (Transformers) have achieved significant success in multivariate time series forecasting (MTSF). However, previous works focus on extracting features either from the time domain or the frequency domain, which inadequately captures the trends and periodic characteristics. To address this issue, we propose a wavelet learning framework to model complex temporal dependencies of the time series data. The wavelet domain integrates both time and frequency information, allowing for the analysis of local characteristics of signals at different scales. Additionally, the Softmax self-attention mechanism used by Transformers has quadratic complexity, which leads to excessive computational costs when capturing long-term dependencies. Therefore, we propose a novel attention mechanism: Rotary Route Attention (RoRA). Unlike Softmax attention, RoRA utilizes rotary position embeddings to inject relative positional information to sequence tokens and introduces a small number of routing tokens $r$ to aggregate information from the $KV$ matrices and redistribute it to the $Q$ matrix, offering linear complexity. We further propose WaveRoRA, which leverages RoRA to capture inter-series dependencies in the wavelet domain. We conduct extensive experiments on eight real-world datasets. The results indicate that WaveRoRA outperforms existing state-of-the-art models while maintaining lower computational costs. Our code is available at https://github.com/Leopold2333/WaveRoRA.

Related papers

• MFRS: A Multi-Frequency Reference Series Approach to Scalable and Accurate Time-Series Forecasting [51.94256702463408]
  Time series predictability is derived from periodic characteristics at different frequencies. We propose a novel time series forecasting method based on multi-frequency reference series correlation analysis. Experiments on major open and synthetic datasets show state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-11T11:40:14Z)
• MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with 0.1K Parameters [6.733646592789575]
  Long-term Time Series Forecasting (LTSF) involves predicting long-term values by analyzing a large amount of historical time-series data to identify patterns and trends. Transformer-based models offer high forecasting accuracy, but they are often too compute-intensive to be deployed on devices with hardware constraints. We propose MixLinear, an ultra-lightweight time series forecasting model specifically designed for resource-constrained devices.
  arXiv  Detail & Related papers  (2024-10-02T23:04:57Z)
• PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting [82.03373838627606]
  Self-attention mechanism in Transformer architecture requires positional embeddings to encode temporal order in time series prediction. We argue that this reliance on positional embeddings restricts the Transformer's ability to effectively represent temporal sequences. We present a model integrating PRE with a standard Transformer encoder, demonstrating state-of-the-art performance on various real-world datasets.
  arXiv  Detail & Related papers  (2024-08-20T01:56:07Z)
• Fourier Controller Networks for Real-Time Decision-Making in Embodied Learning [42.862705980039784]
  Transformer has shown promise in reinforcement learning to model time-varying features. It still suffers from the issues of low data efficiency and high inference latency. In this paper, we propose to investigate the task from a new perspective of the frequency domain.
  arXiv  Detail & Related papers  (2024-05-30T09:43:59Z)
• FAITH: Frequency-domain Attention In Two Horizons for Time Series Forecasting [13.253624747448935]
  Time Series Forecasting plays a crucial role in various fields such as industrial equipment maintenance, meteorology, energy consumption, traffic flow and financial investment. Current deep learning-based predictive models often exhibit a significant deviation between their forecasting outcomes and the ground truth. We propose a novel model Frequency-domain Attention In Two Horizons, which decomposes time series into trend and seasonal components.
  arXiv  Detail & Related papers  (2024-05-22T02:37:02Z)
• ATFNet: Adaptive Time-Frequency Ensembled Network for Long-term Time Series Forecasting [7.694820760102176]
  ATFNet is an innovative framework that combines a time domain module and a frequency domain module. We introduce Dominant Harmonic Series Energy Weighting, a novel mechanism for adjusting the weights between the two modules. Our Complex-valued Spectrum Attention mechanism offers a novel approach to discern the intricate relationships between different frequency combinations.
  arXiv  Detail & Related papers  (2024-04-08T04:41:39Z)
• WFTNet: Exploiting Global and Local Periodicity in Long-term Time Series Forecasting [61.64303388738395]
  We propose a Wavelet-Fourier Transform Network (WFTNet) for long-term time series forecasting. Tests on various time series datasets show WFTNet consistently outperforms other state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-09-20T13:44:18Z)
• W-Transformers : A Wavelet-based Transformer Framework for Univariate Time Series Forecasting [7.075125892721573]
  We build a transformer model for non-stationary time series using wavelet-based transformer encoder architecture. We evaluate our framework on several publicly available benchmark time series datasets from various domains.
  arXiv  Detail & Related papers  (2022-09-08T17:39:38Z)
• HyperTime: Implicit Neural Representation for Time Series [131.57172578210256]
  Implicit neural representations (INRs) have recently emerged as a powerful tool that provides an accurate and resolution-independent encoding of data. In this paper, we analyze the representation of time series using INRs, comparing different activation functions in terms of reconstruction accuracy and training convergence speed. We propose a hypernetwork architecture that leverages INRs to learn a compressed latent representation of an entire time series dataset.
  arXiv  Detail & Related papers  (2022-08-11T14:05:51Z)
• Random Feature Attention [69.4671822971207]
  We propose RFA, a linear time and space attention that uses random feature methods to approximate the softmax function. RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism. Experiments on language modeling and machine translation demonstrate that RFA achieves similar or better performance compared to strong transformer baselines.
  arXiv  Detail & Related papers  (2021-03-03T02:48:56Z)
• Radflow: A Recurrent, Aggregated, and Decomposable Model for Networks of Time Series [77.47313102926017]
  Radflow is a novel model for networks of time series that influence each other. It embodies three key ideas: a recurrent neural network to obtain node embeddings that depend on time, the aggregation of the flow of influence from neighboring nodes with multi-head attention, and the multi-layer decomposition of time series. We show that Radflow can learn different trends and seasonal patterns, that it is robust to missing nodes and edges, and that correlated temporal patterns among network neighbors reflect influence strength.
  arXiv  Detail & Related papers  (2021-02-15T00:57:28Z)
• Deep Cellular Recurrent Network for Efficient Analysis of Time-Series Data with Spatial Information [52.635997570873194]
  This work proposes a novel deep cellular recurrent neural network (DCRNN) architecture to process complex multi-dimensional time series data with spatial information. The proposed architecture achieves state-of-the-art performance while utilizing substantially less trainable parameters when compared to comparable methods in the literature.
  arXiv  Detail & Related papers  (2021-01-12T20:08:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.