Linear Attention is Enough in Spatial-Temporal Forecasting

• URL: http://arxiv.org/abs/2408.09158v2
• Date: Fri, 13 Sep 2024 14:34:26 GMT
• Title: Linear Attention is Enough in Spatial-Temporal Forecasting
• Authors: Xinyu Ning,
• Abstract summary: We propose treating nodes in road networks at different time steps as independent spatial-temporal tokens. We then feed them into a vanilla Transformer to learn complex spatial-temporal patterns. Our code achieves state-of-the-art performance at an affordable computational cost.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As the most representative scenario of spatial-temporal forecasting tasks, the traffic forecasting task attracted numerous attention from machine learning community due to its intricate correlation both in space and time dimension. Existing methods often treat road networks over time as spatial-temporal graphs, addressing spatial and temporal representations independently. However, these approaches struggle to capture the dynamic topology of road networks, encounter issues with message passing mechanisms and over-smoothing, and face challenges in learning spatial and temporal relationships separately. To address these limitations, we propose treating nodes in road networks at different time steps as independent spatial-temporal tokens and feeding them into a vanilla Transformer to learn complex spatial-temporal patterns, design \textbf{STformer} achieving SOTA. Given its quadratic complexity, we introduce a variant \textbf{NSTformer} based on Nystr$\ddot{o}$m method to approximate self-attention with linear complexity but even slightly better than former in a few cases astonishingly. Extensive experimental results on traffic datasets demonstrate that the proposed method achieves state-of-the-art performance at an affordable computational cost. Our code is available at \href{https://github.com/XinyuNing/STformer-and-NSTformer}{https://github.com/XinyuNing/STformer-and-NSTformer}.

Related papers

• DSTCGCN: Learning Dynamic Spatial-Temporal Cross Dependencies for Traffic Forecasting [7.3669718660909735]
  We propose a dynamic spatial-temporal cross graph convolution network to learn dynamic spatial and temporal dependencies jointly. Experiments on six real-world datasets demonstrate that DSTCGCN achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-07-02T08:53:10Z)
• Adaptive Hierarchical SpatioTemporal Network for Traffic Forecasting [70.66710698485745]
  We propose an Adaptive Hierarchical SpatioTemporal Network (AHSTN) to promote traffic forecasting. AHSTN exploits the spatial hierarchy and modeling multi-scale spatial correlations. Experiments on two real-world datasets show that AHSTN achieves better performance over several strong baselines.
  arXiv  Detail & Related papers  (2023-06-15T14:50:27Z)
• Dynamic Graph Convolutional Network with Attention Fusion for Traffic Flow Prediction [10.3426659705376]
  We propose a novel dynamic graph convolution network with attention fusion to model synchronous spatial-temporal correlations. We conduct extensive experiments in four real-world traffic datasets to demonstrate that our method surpasses state-of-the-art performance compared to 18 baseline methods.
  arXiv  Detail & Related papers  (2023-02-24T12:21:30Z)
• PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction [78.05103666987655]
  spatial-temporal Graph Neural Network (GNN) models have emerged as one of the most promising methods to solve this problem. We propose a novel propagation delay-aware dynamic long-range transFormer, namely PDFormer, for accurate traffic flow prediction. Our method can not only achieve state-of-the-art performance but also exhibit competitive computational efficiency.
  arXiv  Detail & Related papers  (2023-01-19T08:42:40Z)
• Spatio-Temporal Joint Graph Convolutional Networks for Traffic Forecasting [75.10017445699532]
  Recent have shifted their focus towards formulating traffic forecasting as atemporal graph modeling problem. We propose a novel approach for accurate traffic forecasting on road networks over multiple future time steps.
  arXiv  Detail & Related papers  (2021-11-25T08:45:14Z)
• Space Meets Time: Local Spacetime Neural Network For Traffic Flow Forecasting [11.495992519252585]
  We argue that such correlations are universal and play a pivotal role in traffic flow. We propose a new spacetime interval learning framework that constructs a local-spacetime context of a traffic sensor. The proposed STNN model can be applied on any unseen traffic networks.
  arXiv  Detail & Related papers  (2021-09-11T09:04:35Z)
• Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting [22.421667339552467]
  Spatial-temporal forecasting has attracted tremendous attention in a wide range of applications, and traffic flow prediction is a canonical and typical example. Existing works typically utilize shallow graph convolution networks (GNNs) and temporal extracting modules to model spatial and temporal dependencies respectively. We propose Spatial-Temporal Graph Ordinary Differential Equation Networks (STGODE), which captures spatial-temporal dynamics through a tensor-based ordinary differential equation (ODE) We evaluate our model on multiple real-world traffic datasets and superior performance is achieved over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-06-24T11:48:45Z)
• Spatial-Temporal Fusion Graph Neural Networks for Traffic Flow Forecasting [35.072979313851235]
  spatial-temporal data forecasting of traffic flow is a challenging task because of complicated spatial dependencies and dynamical trends of temporal pattern between different roads. Existing frameworks typically utilize given spatial adjacency graph and sophisticated mechanisms for modeling spatial and temporal correlations. This paper proposes Spatial-Temporal Fusion Graph Neural Networks (STFGNN) for traffic flow forecasting.
  arXiv  Detail & Related papers  (2020-12-15T14:03:17Z)
• Constructing Geographic and Long-term Temporal Graph for Traffic Forecasting [88.5550074808201]
  We propose Geographic and Long term Temporal Graph Convolutional Recurrent Neural Network (GLT-GCRNN) for traffic forecasting. In this work, we propose a novel framework for traffic forecasting that learns the rich interactions between roads sharing similar geographic or longterm temporal patterns.
  arXiv  Detail & Related papers  (2020-04-23T03:50:46Z)
• A Spatial-Temporal Attentive Network with Spatial Continuity for Trajectory Prediction [74.00750936752418]
  We propose a novel model named spatial-temporal attentive network with spatial continuity (STAN-SC) First, spatial-temporal attention mechanism is presented to explore the most useful and important information. Second, we conduct a joint feature sequence based on the sequence and instant state information to make the generative trajectories keep spatial continuity.
  arXiv  Detail & Related papers  (2020-03-13T04:35:50Z)
• Spatial-Temporal Transformer Networks for Traffic Flow Forecasting [74.76852538940746]
  We propose a novel paradigm of Spatial-Temporal Transformer Networks (STTNs) to improve the accuracy of long-term traffic forecasting. Specifically, we present a new variant of graph neural networks, named spatial transformer, by dynamically modeling directed spatial dependencies. The proposed model enables fast and scalable training over a long range spatial-temporal dependencies.
  arXiv  Detail & Related papers  (2020-01-09T10:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.