VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting

• URL: http://arxiv.org/abs/2403.16536v3
• Date: Sat, 29 Jun 2024 06:23:24 GMT
• Title: VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting
• Authors: Yujin Tang, Peijie Dong, Zhenheng Tang, Xiaowen Chu, Junwei Liang,
• Abstract summary: ViTs or CNNs with RNNs fortemporal forecasting have unparalleled results in predicting temporal and spatial dynamics. Recent Mamba-based architecture has been met with enthusiasm for their exceptional long-sequence modeling capabilities. We propose the VMRNN cell, a recurrent unit that integrates the strengths of Vision Mamba blocks with LSTM.
• Score: 11.058879849373572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Combining CNNs or ViTs, with RNNs for spatiotemporal forecasting, has yielded unparalleled results in predicting temporal and spatial dynamics. However, modeling extensive global information remains a formidable challenge; CNNs are limited by their narrow receptive fields, and ViTs struggle with the intensive computational demands of their attention mechanisms. The emergence of recent Mamba-based architectures has been met with enthusiasm for their exceptional long-sequence modeling capabilities, surpassing established vision models in efficiency and accuracy, which motivates us to develop an innovative architecture tailored for spatiotemporal forecasting. In this paper, we propose the VMRNN cell, a new recurrent unit that integrates the strengths of Vision Mamba blocks with LSTM. We construct a network centered on VMRNN cells to tackle spatiotemporal prediction tasks effectively. Our extensive evaluations show that our proposed approach secures competitive results on a variety of tasks while maintaining a smaller model size. Our code is available at https://github.com/yyyujintang/VMRNN-PyTorch.

Related papers

• Enhancing SNN-based Spatio-Temporal Learning: A Benchmark Dataset and Cross-Modality Attention Model [30.66645039322337]
  High-quality benchmark datasets are great importance to the advances of Artificial Neural Networks (SNNs) Yet, the SNN-based cross-modal fusion remains underexplored. In this work, we present a neuromorphic dataset that can better exploit the inherent-temporal betemporal of SNNs.
  arXiv  Detail & Related papers  (2024-10-21T06:59:04Z)
• Scalable Mechanistic Neural Networks [52.28945097811129]
  We propose an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences. By reformulating the original Mechanistic Neural Network (MNN) we reduce the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear. Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources.
  arXiv  Detail & Related papers  (2024-10-08T14:27:28Z)
• SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models [19.04709216497077]
  We develop spiking state space models (SpikingSSMs) for long sequence learning. Inspired by dendritic neuron structure, we hierarchically integrate neuronal dynamics with the original SSM block. We propose a light-weight surrogate dynamic network which accurately predicts the after-reset membrane potential and compatible to learnable thresholds.
  arXiv  Detail & Related papers  (2024-08-27T09:35:49Z)
• MambaVT: Spatio-Temporal Contextual Modeling for robust RGB-T Tracking [51.28485682954006]
  We propose a pure Mamba-based framework (MambaVT) to fully exploit intrinsic-temporal contextual modeling for robust visible-thermal tracking. Specifically, we devise the long-range cross-frame integration component to globally adapt to target appearance variations. Experiments show the significant potential of vision Mamba for RGB-T tracking, with MambaVT achieving state-of-the-art performance on four mainstream benchmarks.
  arXiv  Detail & Related papers  (2024-08-15T02:29:00Z)
• Deep State Space Recurrent Neural Networks for Time Series Forecasting [0.0]
  This paper introduces novel neural network framework that blend the principles of econometric state space models with the dynamic capabilities of Recurrent Neural Networks (RNNs) According to the results, TKANs, inspired by Kolmogorov-Arnold Networks (KANs) and LSTM, demonstrate promising outcomes.
  arXiv  Detail & Related papers  (2024-07-21T17:59:27Z)
• Efficient and Effective Time-Series Forecasting with Spiking Neural Networks [47.371024581669516]
  Spiking neural networks (SNNs) provide a unique pathway for capturing the intricacies of temporal data. Applying SNNs to time-series forecasting is challenging due to difficulties in effective temporal alignment, complexities in encoding processes, and the absence of standardized guidelines for model selection. We propose a framework for SNNs in time-series forecasting tasks, leveraging the efficiency of spiking neurons in processing temporal information.
  arXiv  Detail & Related papers  (2024-02-02T16:23:50Z)
• Fully Spiking Denoising Diffusion Implicit Models [61.32076130121347]
  Spiking neural networks (SNNs) have garnered considerable attention owing to their ability to run on neuromorphic devices with super-high speeds. We propose a novel approach fully spiking denoising diffusion implicit model (FSDDIM) to construct a diffusion model within SNNs. We demonstrate that the proposed method outperforms the state-of-the-art fully spiking generative model.
  arXiv  Detail & Related papers  (2023-12-04T09:07:09Z)
• Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting [52.47493322446537]
  We develop a adaptive, interpretable and scalable forecasting framework, which seeks to individually model each component of the spatial-temporal patterns. SCNN works with a pre-defined generative process of MTS, which arithmetically characterizes the latent structure of the spatial-temporal patterns. Extensive experiments are conducted to demonstrate that SCNN can achieve superior performance over state-of-the-art models on three real-world datasets.
  arXiv  Detail & Related papers  (2023-05-22T13:39:44Z)
• Batch-Ensemble Stochastic Neural Networks for Out-of-Distribution Detection [55.028065567756066]
  Out-of-distribution (OOD) detection has recently received much attention from the machine learning community due to its importance in deploying machine learning models in real-world applications. In this paper we propose an uncertainty quantification approach by modelling the distribution of features. We incorporate an efficient ensemble mechanism, namely batch-ensemble, to construct the batch-ensemble neural networks (BE-SNNs) and overcome the feature collapse problem. We show that BE-SNNs yield superior performance on several OOD benchmarks, such as the Two-Moons dataset, the FashionMNIST vs MNIST dataset, FashionM
  arXiv  Detail & Related papers  (2022-06-26T16:00:22Z)
• Automatic Remaining Useful Life Estimation Framework with Embedded Convolutional LSTM as the Backbone [5.927250637620123]
  We propose a new LSTM variant called embedded convolutional LSTM (E NeuralTM) In ETM a group of different 1D convolutions is embedded into the LSTM structure. Through this, the temporal information is preserved between and within windows. We show the superiority of our proposed ETM approach over the state-of-the-art approaches on several widely used benchmark data sets for RUL Estimation.
  arXiv  Detail & Related papers  (2020-08-10T08:34:20Z)
• Industrial Forecasting with Exponentially Smoothed Recurrent Neural Networks [0.0]
  We present a class of exponential smoothed recurrent neural networks (RNNs) which are well suited to modeling non-stationary dynamical systems arising in industrial applications. Application of exponentially smoothed RNNs to forecasting electricity load, weather data, and stock prices highlight the efficacy of exponential smoothing of the hidden state for multi-step time series forecasting.
  arXiv  Detail & Related papers  (2020-04-09T17:53:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.