Self-Gating Attention for Efficient Time Series Forecasting

• URL: http://arxiv.org/abs/2607.02344v1
• Date: Thu, 02 Jul 2026 15:49:14 GMT
• Title: Self-Gating Attention for Efficient Time Series Forecasting
• Abstract summary: Self-attention maps in time series forecasting often contain redundant patterns across different timestamps.<n>We propose Self-Gating Attention (SGA), a plug-and-play attention mechanism that represents the attention score with a shared learnable matrix and an input-dependent residual component.
• Score: 32.21325858238185
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer architectures have shown strong potential in time series forecasting, where multi-head self-attention is widely used to capture temporal dependencies across historical timestamps. However, standard self-attention has quadratic time and memory complexity with respect to the look-back length. This cost may limit its use in resource-constrained or high-throughput forecasting systems, where fast and memory-efficient inference is important. Through qualitative and quantitative analyses, we observe that self-attention maps in time series forecasting often contain redundant patterns across different timestamps. This phenomenon can be related to the repeated temporal patterns and relatively stable temporal correlations in many real-world time series. Motivated by this observation, we propose Self-Gating Attention (SGA), a plug-and-play attention mechanism that represents the attention score with a shared learnable matrix and an input-dependent residual component. The shared matrix captures common attention patterns, while the residual component captures input-dependent variations. In this way, SGA avoids the query and key projections used in standard attention score computation, leading to linear time and score-matrix memory complexity with respect to the look-back length. We integrate SGA into several forecasting backbones and compare it with standard self-attention and lightweight attention variants on nine publicly available real-world datasets covering electricity, finance, weather, medical monitoring, human activity, and climate records. The results show that SGA improves inference efficiency on public benchmarks while maintaining competitive forecasting performance against state-of-the-art attention mechanisms. These benchmark results provide deployment-oriented evidence.

Related papers

• GARLIC: Graph Attention-based Relational Learning of Multivariate Time Series in Intensive Care [5.469342914844525]
  We present Attention-based Graph Learning for Intensive Care (GARLIC) model.<n>GARLIC imputes missing data through exponential learnable neural network architecture.<n>On three ICU benchmarks, GARLIC sets the new state of the art in outcome prediction.
  arXiv  Detail & Related papers  (2026-08-11T14:29:48Z)
• TFGformer: Multivariate Time Series Forecasting via Time-Frequency Graph Learning and Covariate Fusion [12.980172248250257]
  We propose a retrieval-augmented forecasting framework that integrates Shape-Aware Memory (SAM) with RevIN normalization for magnitude-robust shape-level retrieval.<n>Experiments on seven public benchmarks show that CrossRAG consistently outperforms both parametric-only baselines and existing retrieval-augmented forecasting methods.
  arXiv  Detail & Related papers  (2026-07-31T14:24:26Z)
• TiWeaver: Unified Temporal Dynamics Modeling via Contextual Patching [15.279995924017458]
  TiWeaver is a unified framework designed to handle temporal dynamics and fine-grained inter-channel dependencies adaptively.<n>We evaluate TiWeaver on 12 real-world time series datasets, where it achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-06-02T04:06:12Z)
• MFRS: A Multi-Frequency Reference Series Approach to Scalable and Accurate Time-Series Forecasting [51.94256702463408]
  Time series predictability is derived from periodic characteristics at different frequencies.<n>We propose a novel time series forecasting method based on multi-frequency reference series correlation analysis.<n> Experiments on major open and synthetic datasets show state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-11T11:40:14Z)
• Powerformer: A Transformer with Weighted Causal Attention for Time-series Forecasting [50.298817606660826]
  We introduce Powerformer, a novel Transformer variant that replaces noncausal attention weights with causal weights that are reweighted according to a smooth heavy-tailed decay.<n>Our empirical results demonstrate that Powerformer achieves state-of-the-art accuracy on public time-series benchmarks.<n>Our analyses show that the model's locality bias is amplified during training, demonstrating an interplay between time-series data and power-law-based attention.
  arXiv  Detail & Related papers  (2025-02-10T04:42:11Z)
• Learning Graph Structures and Uncertainty for Accurate and Calibrated Time-series Forecasting [65.40983982856056]
  We introduce STOIC, that leverages correlations between time-series to learn underlying structure between time-series and to provide well-calibrated and accurate forecasts. Over a wide-range of benchmark datasets STOIC provides 16% more accurate and better-calibrated forecasts.
  arXiv  Detail & Related papers  (2024-07-02T20:14:32Z)
• TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling [67.02157180089573]
  Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. This paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks.
  arXiv  Detail & Related papers  (2024-02-04T13:10:51Z)
• Learning Time-aware Graph Structures for Spatially Correlated Time Series Forecasting [30.93275270960829]
  We propose Time-aware Graph Structure Learning (TagSL), which extracts time-aware correlations among time series. We also present a Graph Convolution-based Gated Recurrent Unit (GCGRU), that jointly captures spatial and temporal dependencies. Finally, we introduce a unified framework named Time-aware Graph Convolutional Recurrent Network (TGCRN), combining TagSL, GCGRU in an encoder-decoder architecture for multi-step-temporal forecasting.
  arXiv  Detail & Related papers  (2023-12-27T04:23:43Z)
• Learning Multi-graph Structure for Temporal Knowledge Graph Reasoning [3.3571415078869955]
  This paper proposes an innovative reasoning approach that focuses on Learning Multi-graph Structure (LMS) LMS incorporates an adaptive gate for merging entity representations both along and across timestamps effectively. It also integrates timestamp semantics into graph attention calculations and time-aware decoders.
  arXiv  Detail & Related papers  (2023-12-04T08:23:09Z)
• Grouped self-attention mechanism for a memory-efficient Transformer [64.0125322353281]
  Real-world tasks such as forecasting weather, electricity consumption, and stock market involve predicting data that vary over time. Time-series data are generally recorded over a long period of observation with long sequences owing to their periodic characteristics and long-range dependencies over time. We propose two novel modules, Grouped Self-Attention (GSA) and Compressed Cross-Attention (CCA) Our proposed model efficiently exhibited reduced computational complexity and performance comparable to or better than existing methods.
  arXiv  Detail & Related papers  (2022-10-02T06:58:49Z)
• An Unsupervised Short- and Long-Term Mask Representation for Multivariate Time Series Anomaly Detection [2.387411589813086]
  This paper proposes an anomaly detection method based on unsupervised Short- and Long-term Mask Representation learning (SLMR) Experiments show that the performance of our method outperforms other state-of-the-art models on three real-world datasets.
  arXiv  Detail & Related papers  (2022-08-19T09:34:11Z)
• Deep Explicit Duration Switching Models for Time Series [84.33678003781908]
  We propose a flexible model that is capable of identifying both state- and time-dependent switching dynamics. State-dependent switching is enabled by a recurrent state-to-switch connection. An explicit duration count variable is used to improve the time-dependent switching behavior.
  arXiv  Detail & Related papers  (2021-10-26T17:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.