ProSTformer: Pre-trained Progressive Space-Time Self-attention Model for Traffic Flow Forecasting

• URL: http://arxiv.org/abs/2111.03459v1
• Date: Wed, 3 Nov 2021 12:20:08 GMT
• Title: ProSTformer: Pre-trained Progressive Space-Time Self-attention Model for Traffic Flow Forecasting
• Authors: Xiao Yan, Xianghua Gan, Jingjing Tang, Rui Wang
• Abstract summary: Two issues prevent the approach from being effectively applied in traffic flow forecasting. We first factor the dependencies and then a space-time self-attention mechanism named ProSTformer. ProSTformer performs better or the same on the big scale datasets than six state-of-the-art methods by RMSE.
• Score: 6.35012051925346
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Traffic flow forecasting is essential and challenging to intelligent city management and public safety. Recent studies have shown the potential of convolution-free Transformer approach to extract the dynamic dependencies among complex influencing factors. However, two issues prevent the approach from being effectively applied in traffic flow forecasting. First, it ignores the spatiotemporal structure of the traffic flow videos. Second, for a long sequence, it is hard to focus on crucial attention due to the quadratic times dot-product computation. To address the two issues, we first factorize the dependencies and then design a progressive space-time self-attention mechanism named ProSTformer. It has two distinctive characteristics: (1) corresponding to the factorization, the self-attention mechanism progressively focuses on spatial dependence from local to global regions, on temporal dependence from inside to outside fragment (i.e., closeness, period, and trend), and finally on external dependence such as weather, temperature, and day-of-week; (2) by incorporating the spatiotemporal structure into the self-attention mechanism, each block in ProSTformer highlights the unique dependence by aggregating the regions with spatiotemporal positions to significantly decrease the computation. We evaluate ProSTformer on two traffic datasets, and each dataset includes three separate datasets with big, medium, and small scales. Despite the radically different design compared to the convolutional architectures for traffic flow forecasting, ProSTformer performs better or the same on the big scale datasets than six state-of-the-art baseline methods by RMSE. When pre-trained on the big scale datasets and transferred to the medium and small scale datasets, ProSTformer achieves a significant enhancement and behaves best.

Related papers

• Spatiotemporal Forecasting of Traffic Flow using Wavelet-based Temporal Attention [3.049645421090079]
  This paper proposes a wavelet-based temporal attention model, namely wavelet-based dynamic graph neural network (DS-DSNN) for tackling the traffic forecasting problem. Our proposed ensemble method can better handle dynamic temporal and spatial benchmarks and make reliable long-term forecasts.
  arXiv  Detail & Related papers  (2024-07-05T11:42:39Z)
• Rethinking Spatio-Temporal Transformer for Traffic Prediction:Multi-level Multi-view Augmented Learning Framework [4.773547922851949]
  Traffic is a challenging-temporal forecasting problem that involves highly complex semantic correlations. This paper proposes a Multi-level Multi-view Augmented-temporal Transformer (LVST) for traffic prediction.
  arXiv  Detail & Related papers  (2024-06-17T07:36:57Z)
• AMP: Autoregressive Motion Prediction Revisited with Next Token Prediction for Autonomous Driving [59.94343412438211]
  We introduce the GPT style next token motion prediction into motion prediction. Different from language data which is composed of homogeneous units -words, the elements in the driving scene could have complex spatial-temporal and semantic relations. We propose to adopt three factorized attention modules with different neighbors for information aggregation and different position encoding styles to capture their relations.
  arXiv  Detail & Related papers  (2024-03-20T06:22:37Z)
• Rethinking Urban Mobility Prediction: A Super-Multivariate Time Series Forecasting Approach [71.67506068703314]
  Long-term urban mobility predictions play a crucial role in the effective management of urban facilities and services. Traditionally, urban mobility data has been structured as videos, treating longitude and latitude as fundamental pixels. In our research, we introduce a fresh perspective on urban mobility prediction. Instead of oversimplifying urban mobility data as traditional video data, we regard it as a complex time series.
  arXiv  Detail & Related papers  (2023-12-04T07:39:05Z)
• Self-Supervised Deconfounding Against Spatio-Temporal Shifts: Theory and Modeling [48.09863133371918]
  In this work, we formalize the problem by constructing a causal graph of past traffic data, future traffic data, and external ST contexts. We show that the failure of prior arts in OOD traffic data is due to ST contexts acting as a confounder, i.e., the common cause for past data and future ones. We devise a Spatio-Temporal sElf-superVised dEconfounding (STEVE) framework to encode traffic data into two disentangled representations for associating invariant and variant ST contexts.
  arXiv  Detail & Related papers  (2023-11-21T09:33:13Z)
• Multi-Scale Spatial-Temporal Recurrent Networks for Traffic Flow Prediction [13.426775574655135]
  We propose a Multi-Scale Spatial-Temporal Recurrent Network for traffic flow prediction, namely MSSTRN. We propose a spatial-temporal synchronous attention mechanism that integrates adaptive position graph convolutions into the self-attention mechanism to achieve synchronous capture of spatial-temporal dependencies. Our model achieves the best prediction accuracy with non-trivial margins compared to all the twenty baseline methods.
  arXiv  Detail & Related papers  (2023-10-12T08:52:36Z)
• Attention-based Spatial-Temporal Graph Convolutional Recurrent Networks for Traffic Forecasting [12.568905377581647]
  Traffic forecasting is one of the most fundamental problems in transportation science and artificial intelligence. Existing methods cannot accurately model both long-term and short-term temporal correlations simultaneously. We propose a novel spatial-temporal neural network framework, which consists of a graph convolutional recurrent module (GCRN) and a global attention module.
  arXiv  Detail & Related papers  (2023-02-25T03:37:00Z)
• PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction [78.05103666987655]
  spatial-temporal Graph Neural Network (GNN) models have emerged as one of the most promising methods to solve this problem. We propose a novel propagation delay-aware dynamic long-range transFormer, namely PDFormer, for accurate traffic flow prediction. Our method can not only achieve state-of-the-art performance but also exhibit competitive computational efficiency.
  arXiv  Detail & Related papers  (2023-01-19T08:42:40Z)
• A Spatial-Temporal Attentive Network with Spatial Continuity for Trajectory Prediction [74.00750936752418]
  We propose a novel model named spatial-temporal attentive network with spatial continuity (STAN-SC) First, spatial-temporal attention mechanism is presented to explore the most useful and important information. Second, we conduct a joint feature sequence based on the sequence and instant state information to make the generative trajectories keep spatial continuity.
  arXiv  Detail & Related papers  (2020-03-13T04:35:50Z)
• Interpretable Crowd Flow Prediction with Spatial-Temporal Self-Attention [16.49833154469825]
  The most challenging part of predicting crowd flow is to measure the complicated spatial-temporal dependencies. We propose a Spatial-Temporal Self-Attention Network (STSAN) with an ST encoding gate that calculates the entire spatial-temporal representation. Experimental results on traffic and mobile data demonstrate that the proposed method reduces inflow and outflow RMSE by 16% and 8% on the Taxi-NYC dataset.
  arXiv  Detail & Related papers  (2020-02-22T12:43:11Z)
• Spatial-Temporal Transformer Networks for Traffic Flow Forecasting [74.76852538940746]
  We propose a novel paradigm of Spatial-Temporal Transformer Networks (STTNs) to improve the accuracy of long-term traffic forecasting. Specifically, we present a new variant of graph neural networks, named spatial transformer, by dynamically modeling directed spatial dependencies. The proposed model enables fast and scalable training over a long range spatial-temporal dependencies.
  arXiv  Detail & Related papers  (2020-01-09T10:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.