SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM

• URL: http://arxiv.org/abs/2308.09891v2
• Date: Sat, 23 Dec 2023 03:40:47 GMT
• Title: SwinLSTM:Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM
• Authors: Song Tang, Chuang Li, Pu Zhang, RongNian Tang
• Abstract summary: We propose a new recurrent cell ConvwinLSTM, which integrates Swin blocks and the LSTM, an extension that replaces the convolutional structure in ConvwinLSTM with the self-attention. Our competitive experimental results demonstrate that learning global spatial dependencies is more advantageous for models to capture Swinwin dependencies.
• Score: 10.104358712577215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Integrating CNNs and RNNs to capture spatiotemporal dependencies is a prevalent strategy for spatiotemporal prediction tasks. However, the property of CNNs to learn local spatial information decreases their efficiency in capturing spatiotemporal dependencies, thereby limiting their prediction accuracy. In this paper, we propose a new recurrent cell, SwinLSTM, which integrates Swin Transformer blocks and the simplified LSTM, an extension that replaces the convolutional structure in ConvLSTM with the self-attention mechanism. Furthermore, we construct a network with SwinLSTM cell as the core for spatiotemporal prediction. Without using unique tricks, SwinLSTM outperforms state-of-the-art methods on Moving MNIST, Human3.6m, TaxiBJ, and KTH datasets. In particular, it exhibits a significant improvement in prediction accuracy compared to ConvLSTM. Our competitive experimental results demonstrate that learning global spatial dependencies is more advantageous for models to capture spatiotemporal dependencies. We hope that SwinLSTM can serve as a solid baseline to promote the advancement of spatiotemporal prediction accuracy. The codes are publicly available at https://github.com/SongTang-x/SwinLSTM.

Related papers

• Beam Prediction based on Large Language Models [51.45077318268427]
  Millimeter-wave (mmWave) communication is promising for next-generation wireless networks but suffers from significant path loss. Traditional deep learning models, such as long short-term memory (LSTM), enhance beam tracking accuracy however are limited by poor robustness and generalization. In this letter, we use large language models (LLMs) to improve the robustness of beam prediction.
  arXiv  Detail & Related papers  (2024-08-16T12:40:01Z)
• Kernel Corrector LSTM [1.034961673489652]
  We propose a new RW-ML algorithm, Kernel Corrector LSTM (KcLSTM), that replaces the meta-learner of cLSTM with a simpler method: Kernel Smoothing. We empirically evaluate the forecasting accuracy and the training time of the new algorithm and compare it with cLSTM and LSTM.
  arXiv  Detail & Related papers  (2024-04-28T18:44:10Z)
• Extreme-Long-short Term Memory for Time-series Prediction [0.0]
  Long Short-Term Memory (LSTM) is a new type of Recurrent Neural Networks (RNN) In this paper, we propose an advanced LSTM algorithm, the Extreme Long Short-Term Memory (E-LSTM) The new E-LSTM requires only 2 epochs to obtain the results of the 7th epoch traditional LSTM.
  arXiv  Detail & Related papers  (2022-10-15T09:45:48Z)
• Image Classification using Sequence of Pixels [3.04585143845864]
  This study compares sequential image classification methods based on recurrent neural networks. We describe methods based on Long-Short-Term memory(LSTM), bidirectional Long-Short-Term memory(BiLSTM) architectures, etc.
  arXiv  Detail & Related papers  (2022-09-23T09:42:44Z)
• Bayesian Neural Network Language Modeling for Speech Recognition [59.681758762712754]
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. In this paper, an overarching full Bayesian learning framework is proposed to account for the underlying uncertainty in LSTM-RNN and Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-28T17:50:19Z)
• Process Outcome Prediction: CNN vs. LSTM (with Attention) [0.15229257192293202]
  We study the performance of Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) on time series problems. Our findings show that all these neural networks achieve satisfactory to high predictive power. We argue that CNNs' speed, early predictive power and robustness should pave the way for their application in process outcome prediction.
  arXiv  Detail & Related papers  (2021-04-14T15:38:32Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)
• Automatic Remaining Useful Life Estimation Framework with Embedded Convolutional LSTM as the Backbone [5.927250637620123]
  We propose a new LSTM variant called embedded convolutional LSTM (E NeuralTM) In ETM a group of different 1D convolutions is embedded into the LSTM structure. Through this, the temporal information is preserved between and within windows. We show the superiority of our proposed ETM approach over the state-of-the-art approaches on several widely used benchmark data sets for RUL Estimation.
  arXiv  Detail & Related papers  (2020-08-10T08:34:20Z)
• Future Vector Enhanced LSTM Language Model for LVCSR [67.03726018635174]
  This paper proposes a novel enhanced long short-term memory (LSTM) LM using the future vector. Experiments show that, the proposed new LSTM LM gets a better result on BLEU scores for long term sequence prediction. Rescoring using both the new and conventional LSTM LMs can achieve a very large improvement on the word error rate.
  arXiv  Detail & Related papers  (2020-07-31T08:38:56Z)
• Object Tracking through Residual and Dense LSTMs [67.98948222599849]
  Deep learning-based trackers based on LSTMs (Long Short-Term Memory) recurrent neural networks have emerged as a powerful alternative. DenseLSTMs outperform Residual and regular LSTM, and offer a higher resilience to nuisances. Our case study supports the adoption of residual-based RNNs for enhancing the robustness of other trackers.
  arXiv  Detail & Related papers  (2020-06-22T08:20:17Z)
• Depth-Adaptive Graph Recurrent Network for Text Classification [71.20237659479703]
  Sentence-State LSTM (S-LSTM) is a powerful and high efficient graph recurrent network. We propose a depth-adaptive mechanism for the S-LSTM, which allows the model to learn how many computational steps to conduct for different words as required.
  arXiv  Detail & Related papers  (2020-02-29T03:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.