Revealing the Power of Masked Autoencoders in Traffic Forecasting

• URL: http://arxiv.org/abs/2309.15169v2
• Date: Sun, 28 Jul 2024 22:41:20 GMT
• Title: Revealing the Power of Masked Autoencoders in Traffic Forecasting
• Authors: Jiarui Sun, Yujie Fan, Chin-Chia Michael Yeh, Wei Zhang, Girish Chowdhary,
• Abstract summary: We propose a plug-and-play framework designed to enhance existing spatial-temporal models on traffic prediction. STMAE consists of two learning stages. In the pretraining stage, an encoder processes partially visible traffic data produced by a dual-masking strategy. Two decoders aim to reconstruct the masked counterparts from both spatial and temporal perspectives. Our results on traffic benchmarks show that STMAE can largely enhance the forecasting capabilities of various spatial-temporal models.
• Score: 16.69508205120188
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traffic forecasting, crucial for urban planning, requires accurate predictions of spatial-temporal traffic patterns across urban areas. Existing research mainly focuses on designing complex models that capture spatial-temporal dependencies among variables explicitly. However, this field faces challenges related to data scarcity and model stability, which results in limited performance improvement. To address these issues, we propose Spatial-Temporal Masked AutoEncoders (STMAE), a plug-and-play framework designed to enhance existing spatial-temporal models on traffic prediction. STMAE consists of two learning stages. In the pretraining stage, an encoder processes partially visible traffic data produced by a dual-masking strategy, including biased random walk-based spatial masking and patch-based temporal masking. Subsequently, two decoders aim to reconstruct the masked counterparts from both spatial and temporal perspectives. The fine-tuning stage retains the pretrained encoder and integrates it with decoders from existing backbones to improve forecasting accuracy. Our results on traffic benchmarks show that STMAE can largely enhance the forecasting capabilities of various spatial-temporal models.

Related papers

• SuperFlow++: Enhanced Spatiotemporal Consistency for Cross-Modal Data Pretraining [62.433137130087445]
  SuperFlow++ is a novel framework that integrates pretraining and downstream tasks using consecutive camera pairs. We show that SuperFlow++ outperforms state-of-the-art methods across diverse tasks and driving conditions. With strong generalizability and computational efficiency, SuperFlow++ establishes a new benchmark for data-efficient LiDAR-based perception in autonomous driving.
  arXiv  Detail & Related papers  (2025-03-25T17:59:57Z)
• PreMixer: MLP-Based Pre-training Enhanced MLP-Mixers for Large-scale Traffic Forecasting [30.055634767677823]
  In urban computing, precise and swift forecasting of time series data from traffic networks is crucial. Current research limitations because of inherent inefficiency of model and their unsuitability for large-scale traffic applications due to model complexity. This paper proposes a novel framework, named PreMixer, designed to bridge this gap. It features a predictive model and a pre-training mechanism, both based on the principles of Multi-Layer Perceptrons (MLP) Our framework achieves comparable state-of-theart performance while maintaining high computational efficiency, as verified by extensive experiments on large-scale traffic datasets.
  arXiv  Detail & Related papers  (2024-12-18T08:35:40Z)
• Navigating Spatio-Temporal Heterogeneity: A Graph Transformer Approach for Traffic Forecasting [13.309018047313801]
  Traffic forecasting has emerged as a crucial research area in the development of smart cities. Recent advancements in network modeling for most-temporal correlations are starting to see diminishing returns in performance. To tackle these challenges, we introduce the Spatio-Temporal Graph Transformer (STGormer) We design two straightforward yet effective spatial encoding methods based on the structure and integrate time position into the vanilla transformer to capture-temporal traffic patterns.
  arXiv  Detail & Related papers  (2024-08-20T13:18:21Z)
• ST-Mamba: Spatial-Temporal Selective State Space Model for Traffic Flow Prediction [32.44888387725925]
  The proposed ST-Mamba model is first to leverage the power of spatial-temporal learning in traffic flow prediction without using graph modeling. The proposed ST-Mamba model achieves a 61.11% improvement in computational speed and increases prediction accuracy by 0.67%. Experiments with real-world traffic datasets demonstrate that the textsfST-Mamba model sets a new benchmark in traffic flow prediction.
  arXiv  Detail & Related papers  (2024-04-20T03:57:57Z)
• Spatial-Temporal Large Language Model for Traffic Prediction [21.69991612610926]
  We propose a Spatial-Temporal Large Language Model (ST-LLM) for traffic prediction. In the ST-LLM, we define timesteps at each location as tokens and design a spatial-temporal embedding to learn the spatial location and global temporal patterns of these tokens. In experiments on real traffic datasets, ST-LLM is a powerful spatial-temporal learner that outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2024-01-18T17:03:59Z)
• Rethinking Urban Mobility Prediction: A Super-Multivariate Time Series Forecasting Approach [71.67506068703314]
  Long-term urban mobility predictions play a crucial role in the effective management of urban facilities and services. Traditionally, urban mobility data has been structured as videos, treating longitude and latitude as fundamental pixels. In our research, we introduce a fresh perspective on urban mobility prediction. Instead of oversimplifying urban mobility data as traditional video data, we regard it as a complex time series.
  arXiv  Detail & Related papers  (2023-12-04T07:39:05Z)
• Transport-Hub-Aware Spatial-Temporal Adaptive Graph Transformer for Traffic Flow Prediction [10.722455633629883]
  We propose a Transport-Hub-aware spatial-temporal adaptive graph transFormer for traffic flow prediction. Specifically, we first design a novel spatial self-attention module to capture the dynamic spatial dependencies. We also employ a temporal self-attention module to detect dynamic temporal patterns in the traffic flow data.
  arXiv  Detail & Related papers  (2023-10-12T13:44:35Z)
• Layout Sequence Prediction From Noisy Mobile Modality [53.49649231056857]
  Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. We propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories.
  arXiv  Detail & Related papers  (2023-10-09T20:32:49Z)
• PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction [78.05103666987655]
  spatial-temporal Graph Neural Network (GNN) models have emerged as one of the most promising methods to solve this problem. We propose a novel propagation delay-aware dynamic long-range transFormer, namely PDFormer, for accurate traffic flow prediction. Our method can not only achieve state-of-the-art performance but also exhibit competitive computational efficiency.
  arXiv  Detail & Related papers  (2023-01-19T08:42:40Z)
• Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction [36.77135502344546]
  We propose a novel Spatio-Supervised Learning (ST-SSL) traffic prediction framework. Our ST-SSL is built over an integrated module with temporal spatial convolutions for encoding the information across space and time. Experiments on four benchmark datasets demonstrate that ST-SSL consistently outperforms various state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-12-07T10:02:01Z)
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery [74.82821342249039]
  We present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE) To leverage temporal information, we include a temporal embedding along with independently masking image patches across time.
  arXiv  Detail & Related papers  (2022-07-17T01:35:29Z)
• P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation [78.83305967085413]
  This paper introduces a novel Pre-trained Spatial Temporal Many-to-One (P-STMO) model for 2D-to-3D human pose estimation task. Our method outperforms state-of-the-art methods with fewer parameters and less computational overhead.
  arXiv  Detail & Related papers  (2022-03-15T04:00:59Z)
• A Spatial-Temporal Attentive Network with Spatial Continuity for Trajectory Prediction [74.00750936752418]
  We propose a novel model named spatial-temporal attentive network with spatial continuity (STAN-SC) First, spatial-temporal attention mechanism is presented to explore the most useful and important information. Second, we conduct a joint feature sequence based on the sequence and instant state information to make the generative trajectories keep spatial continuity.
  arXiv  Detail & Related papers  (2020-03-13T04:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.