Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging

• URL: http://arxiv.org/abs/2404.11511v1
• Date: Wed, 17 Apr 2024 16:06:29 GMT
• Title: Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging
• Authors: Manasi Muglikar, Siddharth Somasundaram, Akshat Dave, Edoardo Charbon, Ramesh Raskar, Davide Scaramuzza,
• Abstract summary: Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements.
• Score: 25.13346470561497
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras due to their desirable properties. SPADs are capable of single-photon sensitivity with microsecond temporal resolution, and event cameras can measure brightness changes up to 1 MHz with low bandwidth requirements. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements. We introduce a sensor fusion framework to combine SPADs with event cameras to improves the reconstruction of high-speed, low-light scenes while reducing the high bandwidth cost associated with using every SPAD frame. Our evaluation, on both synthetic and real sensor data, demonstrates significant enhancements ( > 5 dB PSNR) in reconstructing low-light scenes at high temporal resolution (100 kHz) compared to conventional cameras. Event-SPAD fusion shows great promise for real-world applications, such as robotics or medical imaging.

Related papers

• Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions [56.84882059011291]
  We propose Deblur e-NeRF, a novel method to reconstruct blur-minimal NeRFs from motion-red events. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.
  arXiv  Detail & Related papers  (2024-09-26T15:57:20Z)
• Generalized Event Cameras [15.730999915036705]
  Event cameras capture the world at high time resolution and with minimal bandwidth requirements. We design generalized event cameras that inherently preserve scene intensity in a bandwidth-efficient manner. Our single-photon event cameras are capable of high-speed, high-fidelity imaging at low readout rates.
  arXiv  Detail & Related papers  (2024-07-02T21:48:32Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• Panoramas from Photons [22.437940699523082]
  We present a method capable of estimating extreme scene motion under challenging conditions, such as low light or high dynamic range. Our method relies on grouping and aggregating frames after-the-fact, in a stratified manner. We demonstrate the creation of high-quality panoramas under fast motion and extremely low light, and super-resolution results using a custom single-photon camera prototype.
  arXiv  Detail & Related papers  (2023-09-07T16:07:31Z)
• Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor [58.305341034419136]
  We present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. We propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results.
  arXiv  Detail & Related papers  (2023-08-28T07:56:13Z)
• Make Explicit Calibration Implicit: Calibrate Denoiser Instead of the Noise Model [83.9497193551511]
  We introduce Lighting Every Darkness (LED), which is effective regardless of the digital gain or the camera sensor. LED eliminates the need for explicit noise model calibration, instead utilizing an implicit fine-tuning process that allows quick deployment and requires minimal data. LED also allows researchers to focus more on deep learning advancements while still utilizing sensor engineering benefits.
  arXiv  Detail & Related papers  (2023-08-07T10:09:11Z)
• A Linear Comb Filter for Event Flicker Removal [11.731970880432563]
  Event cameras are bio-inspired sensors that capture per-pixel asynchronous intensity change. Due to their high temporal resolution, event cameras are particularly sensitive to flicker such as from fluorescent or LED lights. We propose a novel linear filter to preprocess event data to remove unwanted flicker events from an event stream.
  arXiv  Detail & Related papers  (2022-05-17T04:47:26Z)
• Are High-Resolution Event Cameras Really Needed? [62.70541164894224]
  In low-illumination conditions and at high speeds, low-resolution cameras can outperform high-resolution ones, while requiring a significantly lower bandwidth. We provide both empirical and theoretical evidence for this claim, which indicates that high-resolution event cameras exhibit higher per-pixel event rates. In most cases, high-resolution event cameras show a lower task performance, compared to lower resolution sensors in these conditions.
  arXiv  Detail & Related papers  (2022-03-28T12:06:20Z)
• ESL: Event-based Structured Light [62.77144631509817]
  Event cameras are bio-inspired sensors providing significant advantages over standard cameras. We propose a novel structured-light system using an event camera to tackle the problem of accurate and high-speed depth sensing.
  arXiv  Detail & Related papers  (2021-11-30T15:47:39Z)
• Quanta Burst Photography [15.722085082004934]
  Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons. We present quanta burst photography, a computational photography technique that leverages SPCs as passive imaging devices for photography in challenging conditions.
  arXiv  Detail & Related papers  (2020-06-21T16:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.