EvConv: Fast CNN Inference on Event Camera Inputs For High-Speed Robot Perception

• URL: http://arxiv.org/abs/2303.04670v1
• Date: Wed, 8 Mar 2023 15:47:13 GMT
• Title: EvConv: Fast CNN Inference on Event Camera Inputs For High-Speed Robot Perception
• Authors: Sankeerth Durvasula, Yushi Guan, Nandita Vijaykumar
• Abstract summary: Event cameras capture visual information with a high temporal resolution and a wide dynamic range. Current convolutional neural network inference on event camera streams cannot currently perform real-time inference at the high speeds at which event cameras operate. This paper presents EvConv, a new approach to enable fast inference on CNNs for inputs from event cameras.
• Score: 1.3869227429939426
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Event cameras capture visual information with a high temporal resolution and a wide dynamic range. This enables capturing visual information at fine time granularities (e.g., microseconds) in rapidly changing environments. This makes event cameras highly useful for high-speed robotics tasks involving rapid motion, such as high-speed perception, object tracking, and control. However, convolutional neural network inference on event camera streams cannot currently perform real-time inference at the high speeds at which event cameras operate - current CNN inference times are typically closer in order of magnitude to the frame rates of regular frame-based cameras. Real-time inference at event camera rates is necessary to fully leverage the high frequency and high temporal resolution that event cameras offer. This paper presents EvConv, a new approach to enable fast inference on CNNs for inputs from event cameras. We observe that consecutive inputs to the CNN from an event camera have only small differences between them. Thus, we propose to perform inference on the difference between consecutive input tensors, or the increment. This enables a significant reduction in the number of floating-point operations required (and thus the inference latency) because increments are very sparse. We design EvConv to leverage the irregular sparsity in increments from event cameras and to retain the sparsity of these increments across all layers of the network. We demonstrate a reduction in the number of floating operations required in the forward pass by up to 98%. We also demonstrate a speedup of up to 1.6X for inference using CNNs for tasks such as depth estimation, object recognition, and optical flow estimation, with almost no loss in accuracy.

Related papers

• EvenNICER-SLAM: Event-based Neural Implicit Encoding SLAM [69.83383687049994]
  We propose EvenNICER-SLAM, a novel approach to dense visual simultaneous localization and mapping. EvenNICER-SLAM incorporates event cameras that respond to intensity changes instead of absolute brightness. Our results suggest the potential for event cameras to improve the robustness of dense SLAM systems against fast camera motion in real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-04T13:52:01Z)
• EV-Catcher: High-Speed Object Catching Using Low-latency Event-based Neural Networks [107.62975594230687]
  We demonstrate an application where event cameras excel: accurately estimating the impact location of fast-moving objects. We introduce a lightweight event representation called Binary Event History Image (BEHI) to encode event data at low latency. We show that the system is capable of achieving a success rate of 81% in catching balls targeted at different locations, with a velocity of up to 13 m/s even on compute-constrained embedded platforms.
  arXiv  Detail & Related papers  (2023-04-14T15:23:28Z)
• Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application [2.048335092363435]
  Event cameras provide continuous asynchronous event streams overcoming the frame-rate limitation. We fuse the complementary accuracy and speed advantages of the frame and event-based pipelines to provide high-speed optical flow.
  arXiv  Detail & Related papers  (2022-07-21T19:15:05Z)
• TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation [78.99283105497489]
  Event camera is a new device to enable video at the presence of arbitrarily complex motion. This paper proposes a novel TimeReplayer algorithm to interpolate videos captured by commodity cameras with events.
  arXiv  Detail & Related papers  (2022-03-25T18:57:42Z)
• ESL: Event-based Structured Light [62.77144631509817]
  Event cameras are bio-inspired sensors providing significant advantages over standard cameras. We propose a novel structured-light system using an event camera to tackle the problem of accurate and high-speed depth sensing.
  arXiv  Detail & Related papers  (2021-11-30T15:47:39Z)
• Reducing the Sim-to-Real Gap for Event Cameras [64.89183456212069]
  Event cameras are paradigm-shifting novel sensors that report asynchronous, per-pixel brightness changes called 'events' with unparalleled low latency. Recent work has demonstrated impressive results using Convolutional Neural Networks (CNNs) for video reconstruction and optic flow with events. We present strategies for improving training data for event based CNNs that result in 20-40% boost in performance of existing video reconstruction networks.
  arXiv  Detail & Related papers  (2020-03-20T02:44:29Z)
• Event-Based Angular Velocity Regression with Spiking Networks [51.145071093099396]
  Spiking Neural Networks (SNNs) process information conveyed as temporal spikes rather than numeric values. We propose, for the first time, a temporal regression problem of numerical values given events from an event camera. We show that we can successfully train an SNN to perform angular velocity regression.
  arXiv  Detail & Related papers  (2020-03-05T17:37:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.