Optical flow estimation from event-based cameras and spiking neural
networks
- URL: http://arxiv.org/abs/2302.06492v2
- Date: Wed, 17 May 2023 13:38:54 GMT
- Title: Optical flow estimation from event-based cameras and spiking neural
networks
- Authors: Javier Cuadrado, Ulysse Ran\c{c}on, Beno\^it Cottereau, Francisco
Barranco and Timoth\'ee Masquelier
- Abstract summary: Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs)
We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations.
Thanks to separable convolutions, we have been able to develop a light model that can nonetheless yield reasonably accurate optical flow estimates.
- Score: 0.4899818550820575
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Event-based cameras are raising interest within the computer vision
community. These sensors operate with asynchronous pixels, emitting events, or
"spikes", when the luminance change at a given pixel since the last event
surpasses a certain threshold. Thanks to their inherent qualities, such as
their low power consumption, low latency and high dynamic range, they seem
particularly tailored to applications with challenging temporal constraints and
safety requirements. Event-based sensors are an excellent fit for Spiking
Neural Networks (SNNs), since the coupling of an asynchronous sensor with
neuromorphic hardware can yield real-time systems with minimal power
requirements. In this work, we seek to develop one such system, using both
event sensor data from the DSEC dataset and spiking neural networks to estimate
optical flow for driving scenarios. We propose a U-Net-like SNN which, after
supervised training, is able to make dense optical flow estimations. To do so,
we encourage both minimal norm for the error vector and minimal angle between
ground-truth and predicted flow, training our model with back-propagation using
a surrogate gradient. In addition, the use of 3d convolutions allows us to
capture the dynamic nature of the data by increasing the temporal receptive
fields. Upsampling after each decoding stage ensures that each decoder's output
contributes to the final estimation. Thanks to separable convolutions, we have
been able to develop a light model (when compared to competitors) that can
nonetheless yield reasonably accurate optical flow estimates.
Related papers
- SDformerFlow: Spatiotemporal swin spikeformer for event-based optical flow estimation [10.696635172502141]
Event cameras generate asynchronous and sparse event streams capturing changes in light intensity.
Spiking neural networks (SNNs) share similar asynchronous and sparse characteristics and are well-suited for event cameras.
We propose two solutions for fast and robust optical flow estimation for event cameras: STTFlowNet and SDFlowformer.
arXiv Detail & Related papers (2024-09-06T07:48:18Z) - A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillation [3.355813093377501]
Event cameras operate differently from traditional digital cameras, continuously capturing data and generating binary spikes that encode time, location, and light intensity.
This necessitates the development of innovative, spike-aware algorithms tailored for event cameras.
We propose a purely spike-driven spike transformer network for depth estimation from spiking camera data.
arXiv Detail & Related papers (2024-04-26T11:32:53Z) - Low-power event-based face detection with asynchronous neuromorphic
hardware [2.0774873363739985]
We present the first instance of an on-chip spiking neural network for event-based face detection deployed on the SynSense Speck neuromorphic chip.
We show how to reduce precision discrepancies between off-chip clock-driven simulation used for training and on-chip event-driven inference.
We achieve an on-chip face detection mAP[0.5] of 0.6 while consuming only 20 mW.
arXiv Detail & Related papers (2023-12-21T19:23:02Z) - Event-based Temporally Dense Optical Flow Estimation with Sequential
Learning [11.026299772309796]
Event cameras capture fast-moving objects without a motion blur.
We show that a temporally dense flow estimation at 100Hz can be achieved by treating the flow estimation as a problem.
arXiv Detail & Related papers (2022-10-03T21:50:14Z) - NAF: Neural Attenuation Fields for Sparse-View CBCT Reconstruction [79.13750275141139]
This paper proposes a novel and fast self-supervised solution for sparse-view CBCT reconstruction.
The desired attenuation coefficients are represented as a continuous function of 3D spatial coordinates, parameterized by a fully-connected deep neural network.
A learning-based encoder entailing hash coding is adopted to help the network capture high-frequency details.
arXiv Detail & Related papers (2022-09-29T04:06:00Z) - Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for
Event-Based Vision [64.71260357476602]
Event-based vision sensors encode local pixel-wise brightness changes in streams of events rather than image frames.
Recent progress in object recognition from event-based sensors has come from conversions of deep neural networks.
We propose a hybrid architecture for end-to-end training of deep neural networks for event-based pattern recognition and object detection.
arXiv Detail & Related papers (2021-12-06T23:45:58Z) - SCFlow: Optical Flow Estimation for Spiking Camera [50.770803466875364]
Spiking camera has enormous potential in real applications, especially for motion estimation in high-speed scenes.
Optical flow estimation has achieved remarkable success in image-based and event-based vision, but % existing methods cannot be directly applied in spike stream from spiking camera.
This paper presents, SCFlow, a novel deep learning pipeline for optical flow estimation for spiking camera.
arXiv Detail & Related papers (2021-10-08T06:16:45Z) - Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor
Fusion and Deep Fused Spiking-Analog Network Architectures [7.565038387344594]
We present a sensor fusion framework for energy-efficient optical flow estimation using both frame- and event-based sensors.
Our network is end-to-end trained using unsupervised learning to avoid expensive video annotations.
arXiv Detail & Related papers (2021-03-19T02:03:33Z) - Combining Events and Frames using Recurrent Asynchronous Multimodal
Networks for Monocular Depth Prediction [51.072733683919246]
We introduce Recurrent Asynchronous Multimodal (RAM) networks to handle asynchronous and irregular data from multiple sensors.
Inspired by traditional RNNs, RAM networks maintain a hidden state that is updated asynchronously and can be queried at any time to generate a prediction.
We show an improvement over state-of-the-art methods by up to 30% in terms of mean depth absolute error.
arXiv Detail & Related papers (2021-02-18T13:24:35Z) - Learning Monocular Dense Depth from Events [53.078665310545745]
Event cameras produce brightness changes in the form of a stream of asynchronous events instead of intensity frames.
Recent learning-based approaches have been applied to event-based data, such as monocular depth prediction.
We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods.
arXiv Detail & Related papers (2020-10-16T12:36:23Z) - Event-based Asynchronous Sparse Convolutional Networks [54.094244806123235]
Event cameras are bio-inspired sensors that respond to per-pixel brightness changes in the form of asynchronous and sparse "events"
We present a general framework for converting models trained on synchronous image-like event representations into asynchronous models with identical output.
We show both theoretically and experimentally that this drastically reduces the computational complexity and latency of high-capacity, synchronous neural networks.
arXiv Detail & Related papers (2020-03-20T08:39:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.