Ultrafast vision perception by neuromorphic optical flow

• URL: http://arxiv.org/abs/2409.15345v1
• Date: Tue, 10 Sep 2024 10:59:32 GMT
• Title: Ultrafast vision perception by neuromorphic optical flow
• Authors: Shengbo Wang, Shuo Gao, Tongming Pu, Liangbing Zhao, Arokia Nathan,
• Abstract summary: 3D neuromorphic optical flow method embeds external motion features directly into hardware. In our demonstration, this approach reduces visual data processing time by an average of 0.3 seconds. Neuromorphic optical flow algorithm's flexibility allows seamless integration with existing algorithms.
• Score: 1.1980928503177917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optical flow is crucial for robotic visual perception, yet current methods primarily operate in a 2D format, capturing movement velocities only in horizontal and vertical dimensions. This limitation results in incomplete motion cues, such as missing regions of interest or detailed motion analysis of different regions, leading to delays in processing high-volume visual data in real-world settings. Here, we report a 3D neuromorphic optical flow method that leverages the time-domain processing capability of memristors to embed external motion features directly into hardware, thereby completing motion cues and dramatically accelerating the computation of movement velocities and subsequent task-specific algorithms. In our demonstration, this approach reduces visual data processing time by an average of 0.3 seconds while maintaining or improving the accuracy of motion prediction, object tracking, and object segmentation. Interframe visual processing is achieved for the first time in UAV scenarios. Furthermore, the neuromorphic optical flow algorithm's flexibility allows seamless integration with existing algorithms, ensuring broad applicability. These advancements open unprecedented avenues for robotic perception, without the trade-off between accuracy and efficiency.

Related papers

• Neuromorphic Optical Flow and Real-time Implementation with Event Cameras [47.11134388304464]
  We build on the latest developments in event-based vision and spiking neural networks. We propose a new network architecture that improves the state-of-the-art self-supervised optical flow accuracy. We demonstrate high speed optical flow prediction with almost two orders of magnitude reduced complexity.
  arXiv  Detail & Related papers  (2023-04-14T14:03:35Z)
• Correlating sparse sensing for large-scale traffic speed estimation: A Laplacian-enhanced low-rank tensor kriging approach [76.45949280328838]
  We propose a Laplacian enhanced low-rank tensor (LETC) framework featuring both lowrankness and multi-temporal correlations for large-scale traffic speed kriging. We then design an efficient solution algorithm via several effective numeric techniques to scale up the proposed model to network-wide kriging.
  arXiv  Detail & Related papers  (2022-10-21T07:25:57Z)
• Time-lapse image classification using a diffractive neural network [0.0]
  We show for the first time a time-lapse image classification scheme using a diffractive network. We show a blind testing accuracy of 62.03% on the optical classification of objects from the CIFAR-10 dataset. This constitutes the highest inference accuracy achieved so far using a single diffractive network.
  arXiv  Detail & Related papers  (2022-08-23T08:16:30Z)
• Motion-aware Memory Network for Fast Video Salient Object Detection [15.967509480432266]
  We design a space-time memory (STM)-based network, which extracts useful temporal information of the current frame from adjacent frames as the temporal branch of VSOD. In the encoding stage, we generate high-level temporal features by using high-level features from the current and its adjacent frames. In the decoding stage, we propose an effective fusion strategy for spatial and temporal branches. The proposed model does not require optical flow or other preprocessing, and can reach a speed of nearly 100 FPS during inference.
  arXiv  Detail & Related papers  (2022-08-01T15:56:19Z)
• Ultra-low Latency Spiking Neural Networks with Spatio-Temporal Compression and Synaptic Convolutional Block [4.081968050250324]
  Spiking neural networks (SNNs) have neuro-temporal information capability, low processing feature, and high biological plausibility. Neuro-MNIST, CIFAR10-S, DVS128 gesture datasets need to aggregate individual events into frames with a higher temporal resolution for event stream classification. We propose a processing-temporal compression method to aggregate individual events into a few time steps of NIST current to reduce the training and inference latency.
  arXiv  Detail & Related papers  (2022-03-18T15:14:13Z)
• Adaptive Latent Space Tuning for Non-Stationary Distributions [62.997667081978825]
  We present a method for adaptive tuning of the low-dimensional latent space of deep encoder-decoder style CNNs. We demonstrate our approach for predicting the properties of a time-varying charged particle beam in a particle accelerator.
  arXiv  Detail & Related papers  (2021-05-08T03:50:45Z)
• Reinforcement Learning with Latent Flow [78.74671595139613]
  Flow of Latents for Reinforcement Learning (Flare) is a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare recovers optimal performance in state-based RL without explicit access to the state velocity. We also show that Flare achieves state-of-the-art performance on pixel-based challenging continuous control tasks within the DeepMind control benchmark suite.
  arXiv  Detail & Related papers  (2021-01-06T03:50:50Z)
• DS-Net: Dynamic Spatiotemporal Network for Video Salient Object Detection [78.04869214450963]
  We propose a novel dynamic temporal-temporal network (DSNet) for more effective fusion of temporal and spatial information. We show that the proposed method achieves superior performance than state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2020-12-09T06:42:30Z)
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance [60.75488333935592]
  Most state-of-the-art methods heavily rely on dense optical flow as motion representation. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. We design a novel motion cue called Persistence of Appearance (PA) In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries.
  arXiv  Detail & Related papers  (2020-08-08T07:09:54Z)
• A Neuromorphic Proto-Object Based Dynamic Visual Saliency Model with an FPGA Implementation [1.2387676601792899]
  We present a neuromorphic, bottom-up, dynamic visual saliency model based on the notion of proto-objects. This model outperforms state-of-the-art dynamic visual saliency models in predicting human eye fixations on a commonly used video dataset. We introduce a Field-Programmable Gate Array implementation of the model on an Opal Kelly 7350 Kintex-7 board.
  arXiv  Detail & Related papers  (2020-02-27T03:31:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.