ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition

• URL: http://arxiv.org/abs/2409.01564v1
• Date: Tue, 3 Sep 2024 03:01:25 GMT
• Title: ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition
• Authors: Shiting Xiao, Yuhang Li, Youngeun Kim, Donghyun Lee, Priyadarshini Panda,
• Abstract summary: Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) We propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost.
• Score: 26.7175155847563
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.

Related papers

• Advancing Spiking Neural Networks towards Multiscale Spatiotemporal Interaction Learning [10.702093960098106]
  Spiking Neural Networks (SNNs) serve as an energy-efficient alternative to Artificial Neural Networks (ANNs) We have designed a Spiking Multiscale Attention (SMA) module that captures multiscaletemporal interaction information. Our approach has achieved state-of-the-art results on mainstream neural datasets.
  arXiv  Detail & Related papers  (2024-05-22T14:16:05Z)
• A Hybrid SNN-ANN Network for Event-based Object Detection with Spatial and Temporal Attention [2.5075774828443467]
  Event cameras offer high temporal resolution and dynamic range with minimal motion blur, making them promising for object detection tasks. While Spiking Neural Networks (SNNs) are a natural match for event-based sensory data, Artificial Neural Networks (ANNs) tend to display more stable training dynamics. We introduce the first Hybrid Attention-based SNN-ANN backbone for object detection using event cameras.
  arXiv  Detail & Related papers  (2024-03-15T10:28:31Z)
• LC-TTFS: Towards Lossless Network Conversion for Spiking Neural Networks with TTFS Coding [55.64533786293656]
  We show that our algorithm can achieve a near-perfect mapping between the activation values of an ANN and the spike times of an SNN on a number of challenging AI tasks. The study paves the way for deploying ultra-low-power TTFS-based SNNs on power-constrained edge computing platforms.
  arXiv  Detail & Related papers  (2023-10-23T14:26:16Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Event-based Video Reconstruction via Potential-assisted Spiking Neural Network [48.88510552931186]
  Bio-inspired neural networks can potentially lead to greater computational efficiency on event-driven hardware. We propose a novel Event-based Video reconstruction framework based on a fully Spiking Neural Network (EVSNN) We find that the spiking neurons have the potential to store useful temporal information (memory) to complete such time-dependent tasks.
  arXiv  Detail & Related papers  (2022-01-25T02:05:20Z)
• Neural Architecture Search for Spiking Neural Networks [10.303676184878896]
  Spiking Neural Networks (SNNs) have gained huge attention as a potential energy-efficient alternative to conventional Artificial Neural Networks (ANNs) Most prior SNN methods use ANN-like architectures, which could provide sub-optimal performance for temporal sequence processing of binary information in SNNs. We introduce a novel Neural Architecture Search (NAS) approach for finding better SNN architectures.
  arXiv  Detail & Related papers  (2022-01-23T16:34:27Z)
• Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision [64.71260357476602]
  Event-based vision sensors encode local pixel-wise brightness changes in streams of events rather than image frames. Recent progress in object recognition from event-based sensors has come from conversions of deep neural networks. We propose a hybrid architecture for end-to-end training of deep neural networks for event-based pattern recognition and object detection.
  arXiv  Detail & Related papers  (2021-12-06T23:45:58Z)
• Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression [12.37129078618206]
  Deep spiking neural networks (SNNs) have emerged as a potential alternative to traditional deep learning frameworks. Most SNN training frameworks yield large inference latency which translates to increased spike activity and reduced energy efficiency. This paper presents a non-iterative SNN training technique thatachieves ultra-high compression with reduced spiking activity.
  arXiv  Detail & Related papers  (2021-07-16T18:23:36Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• You Only Spike Once: Improving Energy-Efficient Neuromorphic Inference to ANN-Level Accuracy [51.861168222799186]
  Spiking Neural Networks (SNNs) are a type of neuromorphic, or brain-inspired network. SNNs are sparse, accessing very few weights, and typically only use addition operations instead of the more power-intensive multiply-and-accumulate operations. In this work, we aim to overcome the limitations of TTFS-encoded neuromorphic systems.
  arXiv  Detail & Related papers  (2020-06-03T15:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.