Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment

• URL: http://arxiv.org/abs/2406.14360v2
• Date: Thu, 1 Aug 2024 17:14:17 GMT
• Title: Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment
• Authors: Yunshan Qi, Lin Zhu, Yifan Zhao, Nan Bao, Jia Li,
• Abstract summary: We propose Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to jointly optimize the learnable poses and NeRF parameters. EBAD-NeRF can obtain accurate camera trajectory during the exposure time and learn a sharper 3D representations compared to prior works.
• Score: 23.15130387716121
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) achieves impressive 3D representation learning and novel view synthesis results with high-quality multi-view images as input. However, motion blur in images often occurs in low-light and high-speed motion scenes, which significantly degrades the reconstruction quality of NeRF. Previous deblurring NeRF methods struggle to estimate pose and lighting changes during the exposure time, making them unable to accurately model the motion blur. The bio-inspired event camera measuring intensity changes with high temporal resolution makes up this information deficiency. In this paper, we propose Event-driven Bundle Adjustment for Deblurring Neural Radiance Fields (EBAD-NeRF) to jointly optimize the learnable poses and NeRF parameters by leveraging the hybrid event-RGB data. An intensity-change-metric event loss and a photo-metric blur loss are introduced to strengthen the explicit modeling of camera motion blur. Experiments on both synthetic and real-captured data demonstrate that EBAD-NeRF can obtain accurate camera trajectory during the exposure time and learn a sharper 3D representations compared to prior works.

Related papers

• Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions [56.84882059011291]
  We propose Deblur e-NeRF, a novel method to reconstruct blur-minimal NeRFs from motion-red events. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.
  arXiv  Detail & Related papers  (2024-09-26T15:57:20Z)
• E$^3$NeRF: Efficient Event-Enhanced Neural Radiance Fields from Blurry Images [25.304680391243537]
  We propose a novel Efficient Event-Enhanced NeRF (E$3$NeRF) We leverage spatial-temporal information from the event stream to evenly distribute learning attention over temporal blur. Experiments on both synthetic data and real-world data demonstrate that E$3$NeRF can effectively learn a sharp NeRF from blurry images.
  arXiv  Detail & Related papers  (2024-08-03T18:47:31Z)
• Mitigating Motion Blur in Neural Radiance Fields with Events and Frames [21.052912896866953]
  We propose a novel approach to enhance NeRF reconstructions under camera motion by fusing frames and events. We explicitly model the blur formation process, exploiting the event double integral as an additional model-based prior. We show, on synthetic and real data, that the proposed approach outperforms existing deblur NeRFs that use only frames.
  arXiv  Detail & Related papers  (2024-03-28T19:06:37Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• Deformable Neural Radiance Fields using RGB and Event Cameras [65.40527279809474]
  We develop a novel method to model the deformable neural radiance fields using RGB and event cameras. The proposed method uses the asynchronous stream of events and sparse RGB frames. Experiments conducted on both realistically rendered graphics and real-world datasets demonstrate a significant benefit of the proposed method.
  arXiv  Detail & Related papers  (2023-09-15T14:19:36Z)
• BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields [9.744593647024253]
  We present a novel bundle adjusted deblur Neural Radiance Fields (BAD-NeRF) BAD-NeRF can be robust to severe motion blurred images and inaccurate camera poses. Our approach models the physical image formation process of a motion blurred image, and jointly learns the parameters of NeRF.
  arXiv  Detail & Related papers  (2022-11-23T10:53:37Z)
• Deblurred Neural Radiance Field with Physical Scene Priors [6.128295038453101]
  This paper proposes a DP-NeRF framework for blurred images, which is constrained with two physical priors. We present extensive experimental results for synthetic and real scenes with two types of blur: camera motion blur and defocus blur.
  arXiv  Detail & Related papers  (2022-11-22T06:40:53Z)
• AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training [100.33713282611448]
  We conduct the first pilot study on training NeRF with high-resolution data. We propose the corresponding solutions, including marrying the multilayer perceptron with convolutional layers. Our approach is nearly free without introducing obvious training/testing costs.
  arXiv  Detail & Related papers  (2022-11-17T17:22:28Z)
• E-NeRF: Neural Radiance Fields from a Moving Event Camera [83.91656576631031]
  Estimating neural radiance fields (NeRFs) from ideal images has been extensively studied in the computer vision community. We present E-NeRF, the first method which estimates a volumetric scene representation in the form of a NeRF from a fast-moving event camera.
  arXiv  Detail & Related papers  (2022-08-24T04:53:32Z)
• Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations [111.08941206369508]
  We propose Augmented NeRF (Aug-NeRF), which for the first time brings the power of robust data augmentations into regularizing the NeRF training. Our proposal learns to seamlessly blend worst-case perturbations into three distinct levels of the NeRF pipeline. Aug-NeRF effectively boosts NeRF performance in both novel view synthesis and underlying geometry reconstruction.
  arXiv  Detail & Related papers  (2022-07-04T02:27:07Z)
• T\"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis [32.878225196378374]
  We introduce a neural representation based on an image formation model for continuous-wave ToF cameras. We show that this approach improves robustness of dynamic scene reconstruction to erroneous calibration and large motions.
  arXiv  Detail & Related papers  (2021-09-30T17:12:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.