Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions

• URL: http://arxiv.org/abs/2409.17988v1
• Date: Thu, 26 Sep 2024 15:57:20 GMT
• Title: Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions
• Authors: Weng Fei Low, Gim Hee Lee,
• Abstract summary: We propose Deblur e-NeRF, a novel method to reconstruct blur-minimal NeRFs from motion-red events. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.
• Score: 56.84882059011291
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The stark contrast in the design philosophy of an event camera makes it particularly ideal for operating under high-speed, high dynamic range and low-light conditions, where standard cameras underperform. Nonetheless, event cameras still suffer from some amount of motion blur, especially under these challenging conditions, in contrary to what most think. This is attributed to the limited bandwidth of the event sensor pixel, which is mostly proportional to the light intensity. Thus, to ensure that event cameras can truly excel in such conditions where it has an edge over standard cameras, it is crucial to account for event motion blur in downstream applications, especially reconstruction. However, none of the recent works on reconstructing Neural Radiance Fields (NeRFs) from events, nor event simulators, have considered the full effects of event motion blur. To this end, we propose, Deblur e-NeRF, a novel method to directly and effectively reconstruct blur-minimal NeRFs from motion-blurred events generated under high-speed motion or low-light conditions. The core component of this work is a physically-accurate pixel bandwidth model proposed to account for event motion blur under arbitrary speed and lighting conditions. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches. Experiments on real and novel realistically simulated sequences verify our effectiveness. Our code, event simulator and synthetic event dataset will be open-sourced.

Related papers

• EvenNICER-SLAM: Event-based Neural Implicit Encoding SLAM [69.83383687049994]
  We propose EvenNICER-SLAM, a novel approach to dense visual simultaneous localization and mapping. EvenNICER-SLAM incorporates event cameras that respond to intensity changes instead of absolute brightness. Our results suggest the potential for event cameras to improve the robustness of dense SLAM systems against fast camera motion in real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-04T13:52:01Z)
• Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment [23.15130387716121]
  We propose Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to jointly optimize the learnable poses and NeRF parameters. EBAD-NeRF can obtain accurate camera trajectory during the exposure time and learn a sharper 3D representations compared to prior works.
  arXiv  Detail & Related papers  (2024-06-20T14:33:51Z)
• From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization [29.197409507402465]
  We propose a novel per-scene optimization strategy tailored for low-light conditions. Our results demonstrate state-of-the-art performance in low-light environments.
  arXiv  Detail & Related papers  (2024-06-12T11:15:59Z)
• Mitigating Motion Blur in Neural Radiance Fields with Events and Frames [21.052912896866953]
  We propose a novel approach to enhance NeRF reconstructions under camera motion by fusing frames and events. We explicitly model the blur formation process, exploiting the event double integral as an additional model-based prior. We show, on synthetic and real data, that the proposed approach outperforms existing deblur NeRFs that use only frames.
  arXiv  Detail & Related papers  (2024-03-28T19:06:37Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• E-NeRF: Neural Radiance Fields from a Moving Event Camera [83.91656576631031]
  Estimating neural radiance fields (NeRFs) from ideal images has been extensively studied in the computer vision community. We present E-NeRF, the first method which estimates a volumetric scene representation in the form of a NeRF from a fast-moving event camera.
  arXiv  Detail & Related papers  (2022-08-24T04:53:32Z)
• Are High-Resolution Event Cameras Really Needed? [62.70541164894224]
  In low-illumination conditions and at high speeds, low-resolution cameras can outperform high-resolution ones, while requiring a significantly lower bandwidth. We provide both empirical and theoretical evidence for this claim, which indicates that high-resolution event cameras exhibit higher per-pixel event rates. In most cases, high-resolution event cameras show a lower task performance, compared to lower resolution sensors in these conditions.
  arXiv  Detail & Related papers  (2022-03-28T12:06:20Z)
• ESL: Event-based Structured Light [62.77144631509817]
  Event cameras are bio-inspired sensors providing significant advantages over standard cameras. We propose a novel structured-light system using an event camera to tackle the problem of accurate and high-speed depth sensing.
  arXiv  Detail & Related papers  (2021-11-30T15:47:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.