Spatially Constrained Transformer with Efficient Global Relation Modelling for Spatio-Temporal Prediction

• URL: http://arxiv.org/abs/2411.06836v1
• Date: Mon, 11 Nov 2024 10:03:59 GMT
• Title: Spatially Constrained Transformer with Efficient Global Relation Modelling for Spatio-Temporal Prediction
• Authors: Ashutosh Sao, Simon Gottschalk,
• Abstract summary: ST-SampleNet is a transformer-based architecture that combines CNNs with self-attention mechanisms to capture both local and global relations. Our experimental variant achieves a 40% reduction in computational costs with only a marginal compromise in performance, approximately 1%.
• Score: 2.016553603539141
• License:
• Abstract: Accurate spatio-temporal prediction is crucial for the sustainable development of smart cities. However, current approaches often struggle to capture important spatio-temporal relationships, particularly overlooking global relations among distant city regions. Most existing techniques predominantly rely on Convolutional Neural Networks (CNNs) to capture global relations. However, CNNs exhibit neighbourhood bias, making them insufficient for capturing distant relations. To address this limitation, we propose ST-SampleNet, a novel transformer-based architecture that combines CNNs with self-attention mechanisms to capture both local and global relations effectively. Moreover, as the number of regions increases, the quadratic complexity of self-attention becomes a challenge. To tackle this issue, we introduce a lightweight region sampling strategy that prunes non-essential regions and enhances the efficiency of our approach. Furthermore, we introduce a spatially constrained position embedding that incorporates spatial neighbourhood information into the self-attention mechanism, aiding in semantic interpretation and improving the performance of ST-SampleNet. Our experimental evaluation on three real-world datasets demonstrates the effectiveness of ST-SampleNet. Additionally, our efficient variant achieves a 40% reduction in computational costs with only a marginal compromise in performance, approximately 1%.

Related papers

• Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awareness [29.679323144520037]
  Federated Learning (FL) can be coordinated under the orchestration of a central server to build a privacy-preserving model. We propose a novel FL algorithm, FedTOGA, designed to consider generalization objectives while maintaining minimal uplink communication overhead.
  arXiv  Detail & Related papers  (2024-08-26T09:42:18Z)
• SFANet: Spatial-Frequency Attention Network for Weather Forecasting [54.470205739015434]
  Weather forecasting plays a critical role in various sectors, driving decision-making and risk management. Traditional methods often struggle to capture the complex dynamics of meteorological systems. We propose a novel framework designed to address these challenges and enhance the accuracy of weather prediction.
  arXiv  Detail & Related papers  (2024-05-29T08:00:15Z)
• Typhoon Intensity Prediction with Vision Transformer [51.84456610977905]
  We introduce "Typhoon Intensity Transformer" (Tint) to predict typhoon intensity accurately across space and time. Tint uses self-attention mechanisms with global receptive fields per layer. Experiments on a publicly available typhoon benchmark validate the efficacy of Tint.
  arXiv  Detail & Related papers  (2023-11-28T03:11:33Z)
• Multi-Temporal Relationship Inference in Urban Areas [75.86026742632528]
  Finding temporal relationships among locations can benefit a bunch of urban applications, such as dynamic offline advertising and smart public transport planning. We propose a solution to Trial with a graph learning scheme, which includes a spatially evolving graph neural network (SEENet) SEConv performs the intra-time aggregation and inter-time propagation to capture the multifaceted spatially evolving contexts from the view of location message passing. SE-SSL designs time-aware self-supervised learning tasks in a global-local manner with additional evolving constraint to enhance the location representation learning and further handle the relationship sparsity.
  arXiv  Detail & Related papers  (2023-06-15T07:48:32Z)
• SARN: Structurally-Aware Recurrent Network for Spatio-Temporal Disaggregation [8.636014676778682]
  Open data is frequently released spatially aggregated, usually to comply with privacy policies. But coarse, heterogeneous aggregations complicate coherent learning and integration for downstream AI/ML systems. We propose an overarching model named Structurally-Aware Recurrent Network (SARN), which integrates structurally-aware spatial attention layers into the Gated Recurrent Unit (GRU) model. For scenarios with limited historical training data, we show that a model pre-trained on one city variable can be fine-tuned for another city variable using only a few hundred samples.
  arXiv  Detail & Related papers  (2023-06-09T21:01:29Z)
• Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape [59.841889495864386]
  In federated learning (FL), a cluster of local clients are chaired under the coordination of a global server. Clients are prone to overfit into their own optima, which extremely deviates from the global objective. ttfamily FedSMOO adopts a dynamic regularizer to guarantee the local optima towards the global objective. Our theoretical analysis indicates that ttfamily FedSMOO achieves fast $mathcalO (1/T)$ convergence rate with low bound generalization.
  arXiv  Detail & Related papers  (2023-05-19T10:47:44Z)
• FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy [84.45004766136663]
  Federated learning is an emerging distributed machine learning framework. It suffers from the non-vanishing biases introduced by the local inconsistent optimal and the rugged client-drifts by the local over-fitting. We propose a novel and practical method, FedSpeed, to alleviate the negative impacts posed by these problems.
  arXiv  Detail & Related papers  (2023-02-21T03:55:29Z)
• STJLA: A Multi-Context Aware Spatio-Temporal Joint Linear Attention Network for Traffic Forecasting [7.232141271583618]
  We propose a novel deep learning model for traffic forecasting named inefficient-Context Spatio-Temporal Joint Linear Attention (SSTLA) SSTLA applies linear attention to a joint graph to capture global dependence between alltemporal- nodes efficiently. Experiments on two real-world traffic datasets, England and Temporal7, demonstrate that our STJLA can achieve 9.83% and 3.08% 3.08% accuracy in MAE measure over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-12-04T06:39:18Z)
• An Entropy-guided Reinforced Partial Convolutional Network for Zero-Shot Learning [77.72330187258498]
  We propose a novel Entropy-guided Reinforced Partial Convolutional Network (ERPCNet) ERPCNet extracts and aggregates localities based on semantic relevance and visual correlations without human-annotated regions. It not only discovers global-cooperative localities dynamically but also converges faster for policy gradient optimization.
  arXiv  Detail & Related papers  (2021-11-03T11:13:13Z)
• Traffic Flow Forecasting with Spatial-Temporal Graph Diffusion Network [39.65520262751766]
  We develop a new traffic prediction framework-Spatial-Temporal Graph Diffusion Network (ST-GDN) In particular, ST-GDN is a hierarchically structured graph neural architecture which learns not only the local region-wise geographical dependencies, but also the spatial semantics from a global perspective. Experiments on several real-life traffic datasets demonstrate that ST-GDN outperforms different types of state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-10-08T11:19:06Z)
• Space Meets Time: Local Spacetime Neural Network For Traffic Flow Forecasting [11.495992519252585]
  We argue that such correlations are universal and play a pivotal role in traffic flow. We propose a new spacetime interval learning framework that constructs a local-spacetime context of a traffic sensor. The proposed STNN model can be applied on any unseen traffic networks.
  arXiv  Detail & Related papers  (2021-09-11T09:04:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.