Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams

• URL: http://arxiv.org/abs/2412.06770v2
• Date: Tue, 22 Apr 2025 20:42:42 GMT
• Title: Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams
• Authors: Viktor Rudnev, Gereon Fox, Mohamed Elgharib, Christian Theobalt, Vladislav Golyanik,
• Abstract summary: Volumetric reconstruction of dynamic scenes is an important problem in computer vision.<n>It is especially challenging in poor lighting and with fast motion.<n>We propose the first method totemporally reconstruct a scene from sparse multi-view event streams and sparse RGB frames.
• Score: 69.65147723239153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Volumetric reconstruction of dynamic scenes is an important problem in computer vision. It is especially challenging in poor lighting and with fast motion. This is partly due to limitations of RGB cameras: To capture frames under low lighting, the exposure time needs to be increased, which leads to more motion blur. In contrast, event cameras, which record changes in pixel brightness asynchronously, are much less dependent on lighting, making them more suitable for recording fast motion. We hence propose the first method to spatiotemporally reconstruct a scene from sparse multi-view event streams and sparse RGB frames. We train a sequence of cross-faded time-conditioned NeRF models, one per short recording segment. The individual segments are supervised with a set of event- and RGB-based losses and sparse-view regularisation. We assemble a real-world multi-view camera rig with six static event cameras around the object and record a benchmark multi-view event stream dataset of challenging motions. Our work outperforms RGB-based baselines, producing state-of-the-art results, and opens up the topic of multi-view event-based reconstruction as a new path for fast scene capture beyond RGB cameras. The code and the data will be released soon at https://4dqv.mpi-inf.mpg.de/DynEventNeRF/

Related papers

• EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting [76.02450110026747]
  Event cameras, inspired by biological vision, record pixel-wise intensity changes asynchronously with high temporal resolution. We propose Event-Aided Free-Trajectory 3DGS, which seamlessly integrates the advantages of event cameras into 3DGS. We evaluate our method on the public Tanks and Temples benchmark and a newly collected real-world dataset, RealEv-DAVIS.
  arXiv  Detail & Related papers  (2024-10-20T13:44:24Z)
• Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions [56.84882059011291]
  We propose Deblur e-NeRF, a novel method to reconstruct blur-minimal NeRFs from motion-red events. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.
  arXiv  Detail & Related papers  (2024-09-26T15:57:20Z)
• Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction [51.87279764576998]
  We propose EvRGBHand -- the first approach for 3D hand mesh reconstruction with an event camera and an RGB camera compensating for each other. EvRGBHand can tackle overexposure and motion blur issues in RGB-based HMR and foreground scarcity and background overflow issues in event-based HMR.
  arXiv  Detail & Related papers  (2024-03-12T06:04:50Z)
• Event-based Continuous Color Video Decompression from Single Frames [36.4263932473053]
  We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image and an event camera stream.<n>Our approach combines continuous long-range motion modeling with a neural synthesis model, enabling frame prediction at arbitrary times within the events.
  arXiv  Detail & Related papers  (2023-11-30T18:59:23Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• Deformable Convolutions and LSTM-based Flexible Event Frame Fusion Network for Motion Deblurring [7.187030024676791]
  Event cameras differ from conventional RGB cameras in that they produce asynchronous data sequences. While RGB cameras capture every frame at a fixed rate, event cameras only capture changes in the scene, resulting in sparse and asynchronous data output. Recent state-of-the-art CNN-based deblurring solutions produce multiple 2-D event frames based on the accumulation of event data over a time period. It is particularly useful for scenarios in which exposure times vary depending on factors such as lighting conditions or the presence of fast-moving objects in the scene.
  arXiv  Detail & Related papers  (2023-06-01T15:57:12Z)
• Event-based Camera Tracker by $\nabla$t NeRF [11.572930535988325]
  We show that we can recover the camera pose by minimizing the error between sparse events and the temporal gradient of the scene represented as a neural radiance field (NeRF) We propose an event-based camera pose tracking framework called TeGRA which realizes the pose update by using the sparse event's observation.
  arXiv  Detail & Related papers  (2023-04-07T16:03:21Z)
• EventNeRF: Neural Radiance Fields from a Single Colour Event Camera [81.19234142730326]
  This paper proposes the first approach for 3D-consistent, dense and novel view synthesis using just a single colour event stream as input. At its core is a neural radiance field trained entirely in a self-supervised manner from events while preserving the original resolution of the colour event channels. We evaluate our method qualitatively and numerically on several challenging synthetic and real scenes and show that it produces significantly denser and more visually appealing renderings.
  arXiv  Detail & Related papers  (2022-06-23T17:59:53Z)
• Event-Based Dense Reconstruction Pipeline [5.341354397748495]
  Event cameras are a new type of sensors that are different from traditional cameras. Deep learning is used to reconstruct intensity images from events. structure from motion (SfM) is used to estimate camera intrinsic, extrinsic and sparse point cloud.
  arXiv  Detail & Related papers  (2022-03-23T08:37:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.