Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness

• URL: http://arxiv.org/abs/2402.08936v1
• Date: Wed, 14 Feb 2024 04:34:48 GMT
• Title: Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness
• Abstract summary: We propose a temporal attention mechanism to throttle the camera output and pay attention to it only when the visual events cannot be well predicted. We demonstrate that the predictive attention can reduce 46.7% of data communication between the camera and the processor and reduce 43.8% activities in the processor.
• Score: 5.541459858161597
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Dynamic Vision Sensor (DVS) is an innovative technology that efficiently captures and encodes visual information in an event-driven manner. By combining it with event-driven neuromorphic processing, the sparsity in DVS camera output can result in high energy efficiency. However, similar to many embedded systems, the off-chip communication between the camera and processor presents a bottleneck in terms of power consumption. Inspired by the predictive coding model and expectation suppression phenomenon found in human brain, we propose a temporal attention mechanism to throttle the camera output and pay attention to it only when the visual events cannot be well predicted. The predictive attention not only reduces power consumption in the sensor-processor interface but also effectively decreases the computational workload by filtering out noisy events. We demonstrate that the predictive attention can reduce 46.7% of data communication between the camera and the processor and reduce 43.8% computation activities in the processor.

Related papers

• CS3D: An Efficient Facial Expression Recognition via Event Vision [3.2557351161800123]
  Event cameras are becoming more widely adopted as they surpass RGB cameras in capturing facial expression changes.<n>Traditional deep learning methods for facial expression analysis are energy-intensive.<n>We propose the CS3D framework by decomposing the Convolutional 3D method to reduce the computational complexity and energy consumption.
  arXiv  Detail & Related papers  (2025-12-10T12:42:28Z)
• Wandering around: A bioinspired approach to visual attention through object motion sensitivity [40.966228784674115]
  Active vision enables dynamic visual perception, offering an alternative to static feedforward architectures in computer vision. Event-based cameras, inspired by the mammalian retina, enhance this capability by capturing asynchronous scene changes. To distinguish moving objects while the event-based camera is in motion the agent requires an object motion segmentation mechanism. This work presents a Convolutional Neural Network bio-inspired attention system for selective attention through object motion sensitivity.
  arXiv  Detail & Related papers  (2025-02-10T18:16:30Z)
• Event-based vision for egomotion estimation using precise event timing [0.6262316762195913]
  Egomotion estimation is crucial for applications such as autonomous navigation and robotics. Traditional methods relying on inertial sensors are sensitive to external conditions. Vision-based methods provide an efficient alternative by capturing data only when changes are perceived in the scene.
  arXiv  Detail & Related papers  (2025-01-20T15:41:33Z)
• Integration of Communication and Computational Imaging [49.2442836992307]
  We propose a novel framework that integrates communication and computational imaging (ICCI) for remote perception. ICCI framework performs a full-link information transfer optimization, aiming to minimize information loss from the generation of the information source to the execution of the final vision tasks. An 80 km 27-band hyperspectral video perception with a rate of 30 fps is experimentally achieved.
  arXiv  Detail & Related papers  (2024-10-25T09:19:59Z)
• Low Latency Visual Inertial Odometry with On-Sensor Accelerated Optical Flow for Resource-Constrained UAVs [13.037162115493393]
  On-sensor hardware acceleration is a promising approach to enable low latency Visual Inertial Odometry (VIO) This paper assesses the speed-up in a VIO sensor system exploiting a compact OF sensor consisting of a global shutter camera and an Application Specific Integrated Circuit (ASIC) By replacing the feature tracking logic of the VINS-Mono pipeline with data from this OF camera, we demonstrate a 49.4% reduction in latency and a 53.7% reduction of compute load of the VIO pipeline over the original VINS-Mono implementation.
  arXiv  Detail & Related papers  (2024-06-19T08:51:19Z)
• A motion-based compression algorithm for resource-constrained video camera traps [4.349838917565205]
  We introduce a new motion analysis-based video compression algorithm specifically designed for camera traps. The algorithm identifies and stores only image regions depicting motion relevant to motion pollination monitoring. Our experiments demonstrate the algorithm's capability to preserve critical information for insect behaviour analysis.
  arXiv  Detail & Related papers  (2024-05-23T10:39:33Z)
• Efficient and accurate neural field reconstruction using resistive memory [52.68088466453264]
  Traditional signal reconstruction methods on digital computers face both software and hardware challenges. We propose a systematic approach with software-hardware co-optimizations for signal reconstruction from sparse inputs. This work advances the AI-driven signal restoration technology and paves the way for future efficient and robust medical AI and 3D vision applications.
  arXiv  Detail & Related papers  (2024-04-15T09:33:09Z)
• Low-power event-based face detection with asynchronous neuromorphic hardware [2.0774873363739985]
  We present the first instance of an on-chip spiking neural network for event-based face detection deployed on the SynSense Speck neuromorphic chip. We show how to reduce precision discrepancies between off-chip clock-driven simulation used for training and on-chip event-driven inference. We achieve an on-chip face detection mAP[0.5] of 0.6 while consuming only 20 mW.
  arXiv  Detail & Related papers  (2023-12-21T19:23:02Z)
• Pushing the Limits of Asynchronous Graph-based Object Detection with Event Cameras [62.70541164894224]
  We introduce several architecture choices which allow us to scale the depth and complexity of such models while maintaining low computation. Our method runs 3.7 times faster than a dense graph neural network, taking only 8.4 ms per forward pass.
  arXiv  Detail & Related papers  (2022-11-22T15:14:20Z)
• MSS-DepthNet: Depth Prediction with Multi-Step Spiking Neural Network [8.53512216864715]
  Spiking neural network is a novel event-based computational paradigm that is considered to be well suited for processing event camera tasks. This work proposes a spiking neural network architecture with a novel residual block designed and multi-dimension attention modules combined. This model outperforms previous ANN networks of the same size on the MVSEC dataset and shows great computational efficiency.
  arXiv  Detail & Related papers  (2022-11-22T10:35:36Z)
• ETAD: A Unified Framework for Efficient Temporal Action Detection [70.21104995731085]
  Untrimmed video understanding such as temporal action detection (TAD) often suffers from the pain of huge demand for computing resources. We build a unified framework for efficient end-to-end temporal action detection (ETAD) ETAD achieves state-of-the-art performance on both THUMOS-14 and ActivityNet-1.3.
  arXiv  Detail & Related papers  (2022-05-14T21:16:21Z)
• Asynchronous Optimisation for Event-based Visual Odometry [53.59879499700895]
  Event cameras open up new possibilities for robotic perception due to their low latency and high dynamic range. We focus on event-based visual odometry (VO) We propose an asynchronous structure-from-motion optimisation back-end.
  arXiv  Detail & Related papers  (2022-03-02T11:28:47Z)
• BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing [48.11023234245863]
  We propose a new framework called BottleFit, which includes a novel training strategy to achieve high accuracy even with strong compression rates. BottleFit achieves 77.1% data compression with up to 0.6% accuracy loss on ImageNet dataset. We show that BottleFit decreases power consumption and latency respectively by up to 49% and 89% with respect to (w.r.t.) local computing and by 37% and 55% w.r.t. edge offloading.
  arXiv  Detail & Related papers  (2022-01-07T22:08:07Z)
• Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks [40.44712305614071]
  We present Spike-FlowNet, a deep hybrid neural network architecture integrating SNNs and ANNs for efficiently estimating optical flow from sparse event camera outputs. The network is end-to-end trained with self-supervised learning on Multi-Vehicle Stereo Event Camera (MVSEC) dataset.
  arXiv  Detail & Related papers  (2020-03-14T20:37:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.