Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging

• URL: http://arxiv.org/abs/2404.11511v1
• Date: Wed, 17 Apr 2024 16:06:29 GMT
• Title: Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging
• Authors: Manasi Muglikar, Siddharth Somasundaram, Akshat Dave, Edoardo Charbon, Ramesh Raskar, Davide Scaramuzza,
• Abstract summary: Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements.
• Score: 25.13346470561497
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional cameras face a trade-off between low-light performance and high-speed imaging: longer exposure times to capture sufficient light results in motion blur, whereas shorter exposures result in Poisson-corrupted noisy images. While burst photography techniques help mitigate this tradeoff, conventional cameras are fundamentally limited in their sensor noise characteristics. Event cameras and single-photon avalanche diode (SPAD) sensors have emerged as promising alternatives to conventional cameras due to their desirable properties. SPADs are capable of single-photon sensitivity with microsecond temporal resolution, and event cameras can measure brightness changes up to 1 MHz with low bandwidth requirements. We show that these properties are complementary, and can help achieve low-light, high-speed image reconstruction with low bandwidth requirements. We introduce a sensor fusion framework to combine SPADs with event cameras to improves the reconstruction of high-speed, low-light scenes while reducing the high bandwidth cost associated with using every SPAD frame. Our evaluation, on both synthetic and real sensor data, demonstrates significant enhancements ( > 5 dB PSNR) in reconstructing low-light scenes at high temporal resolution (100 kHz) compared to conventional cameras. Event-SPAD fusion shows great promise for real-world applications, such as robotics or medical imaging.

Related papers

• Rethinking High-speed Image Reconstruction Framework with Spike Camera [48.627095354244204]
  Spike cameras generate continuous spike streams to capture high-speed scenes with lower bandwidth and higher dynamic range than traditional RGB cameras. We introduce a novel spike-to-image reconstruction framework SpikeCLIP that goes beyond traditional training paradigms. Our experiments on real-world low-light datasets demonstrate that SpikeCLIP significantly enhances texture details and the luminance balance of recovered images.
  arXiv  Detail & Related papers  (2025-01-08T13:00:17Z)
• A Plug-and-Play Algorithm for 3D Video Super-Resolution of Single-Photon LiDAR data [5.378429123269604]
  Single-photon avalanche diodes (SPADs) are advanced sensors capable of detecting individual photons and recording their arrival times with picosecond resolution. We propose a novel computational imaging algorithm to improve the 3D reconstruction of moving scenes from SPAD data.
  arXiv  Detail & Related papers  (2024-12-12T16:33:06Z)
• Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions [56.84882059011291]
  We propose Deblur e-NeRF, a novel method to reconstruct blur-minimal NeRFs from motion-red events. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.
  arXiv  Detail & Related papers  (2024-09-26T15:57:20Z)
• Generalized Event Cameras [15.730999915036705]
  Event cameras capture the world at high time resolution and with minimal bandwidth requirements. We design generalized event cameras that inherently preserve scene intensity in a bandwidth-efficient manner. Our single-photon event cameras are capable of high-speed, high-fidelity imaging at low readout rates.
  arXiv  Detail & Related papers  (2024-07-02T21:48:32Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• Panoramas from Photons [22.437940699523082]
  We present a method capable of estimating extreme scene motion under challenging conditions, such as low light or high dynamic range. Our method relies on grouping and aggregating frames after-the-fact, in a stratified manner. We demonstrate the creation of high-quality panoramas under fast motion and extremely low light, and super-resolution results using a custom single-photon camera prototype.
  arXiv  Detail & Related papers  (2023-09-07T16:07:31Z)
• Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor [58.305341034419136]
  We present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. We propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results.
  arXiv  Detail & Related papers  (2023-08-28T07:56:13Z)
• Are High-Resolution Event Cameras Really Needed? [62.70541164894224]
  In low-illumination conditions and at high speeds, low-resolution cameras can outperform high-resolution ones, while requiring a significantly lower bandwidth. We provide both empirical and theoretical evidence for this claim, which indicates that high-resolution event cameras exhibit higher per-pixel event rates. In most cases, high-resolution event cameras show a lower task performance, compared to lower resolution sensors in these conditions.
  arXiv  Detail & Related papers  (2022-03-28T12:06:20Z)
• ESL: Event-based Structured Light [62.77144631509817]
  Event cameras are bio-inspired sensors providing significant advantages over standard cameras. We propose a novel structured-light system using an event camera to tackle the problem of accurate and high-speed depth sensing.
  arXiv  Detail & Related papers  (2021-11-30T15:47:39Z)
• Quanta Burst Photography [15.722085082004934]
  Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons. We present quanta burst photography, a computational photography technique that leverages SPCs as passive imaging devices for photography in challenging conditions.
  arXiv  Detail & Related papers  (2020-06-21T16:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.