BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling

• URL: http://arxiv.org/abs/2302.13543v3
• Date: Sat, 25 Mar 2023 02:18:02 GMT
• Title: BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling
• Authors: Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, Anton Van Den Hengel
• Abstract summary: We propose a framework that factorizes time and space by formulating a scene as a composition of bandlimited, high-dimensional signals. We demonstrate compelling results across complex dynamic scenes that involve changes in lighting, texture and long-range dynamics.
• Score: 43.246536947828844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reasoning the 3D structure of a non-rigid dynamic scene from a single moving camera is an under-constrained problem. Inspired by the remarkable progress of neural radiance fields (NeRFs) in photo-realistic novel view synthesis of static scenes, extensions have been proposed for dynamic settings. These methods heavily rely on neural priors in order to regularize the problem. In this work, we take a step back and reinvestigate how current implementations may entail deleterious effects, including limited expressiveness, entanglement of light and density fields, and sub-optimal motion localization. As a remedy, we advocate for a bridge between classic non-rigid-structure-from-motion (\nrsfm) and NeRF, enabling the well-studied priors of the former to constrain the latter. To this end, we propose a framework that factorizes time and space by formulating a scene as a composition of bandlimited, high-dimensional signals. We demonstrate compelling results across complex dynamic scenes that involve changes in lighting, texture and long-range dynamics.

Related papers

• Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling [70.34875558830241]
  We present a way for learning a-temporal (4D) embedding, based on semantic semantic gears to allow for stratified modeling of dynamic regions of rendering the scene. At the same time, almost for free, our tracking approach enables free-viewpoint of interest - a functionality not yet achieved by existing NeRF-based methods.
  arXiv  Detail & Related papers  (2024-06-06T03:37:39Z)
• NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation [76.02304140027087]
  We propose a novel paradigm, namely "Heuristics-Guided harmoniously" (HuGS) HuGS significantly enhances the separation of static scenes from transient distractors by combining the strengths of hand-crafted synthesiss and state-of-the-art segmentation models. Experiments demonstrate the superiority and robustness of our method in mitigating transient distractors for NeRFs trained in non-static scenes.
  arXiv  Detail & Related papers  (2024-03-26T09:42:28Z)
• DyBluRF: Dynamic Neural Radiance Fields from Blurry Monocular Video [18.424138608823267]
  We propose DyBluRF, a dynamic radiance field approach that synthesizes sharp novel views from a monocular video affected by motion blur. To account for motion blur in input images, we simultaneously capture the camera trajectory and object Discrete Cosine Transform (DCT) trajectories within the scene.
  arXiv  Detail & Related papers  (2024-03-15T08:48:37Z)
• NeVRF: Neural Video-based Radiance Fields for Long-duration Sequences [53.8501224122952]
  We propose a novel neural video-based radiance fields (NeVRF) representation. NeVRF marries neural radiance field with image-based rendering to support photo-realistic novel view synthesis on long-duration dynamic inward-looking scenes. Our experiments demonstrate the effectiveness of NeVRF in enabling long-duration sequence rendering, sequential data reconstruction, and compact data storage.
  arXiv  Detail & Related papers  (2023-12-10T11:14:30Z)
• DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting [35.69069478773709]
  We argue that the per-point motions of a dynamic scene can be decomposed into a small set of explicit or learned trajectories. Our representation is interpretable, efficient, and expressive enough to offer real-time view synthesis of complex dynamic scene motions.
  arXiv  Detail & Related papers  (2023-11-30T18:59:11Z)
• OD-NeRF: Efficient Training of On-the-Fly Dynamic Neural Radiance Fields [63.04781030984006]
  Dynamic neural radiance fields (dynamic NeRFs) have demonstrated impressive results in novel view synthesis on 3D dynamic scenes. We propose OD-NeRF to efficiently train and render dynamic NeRFs on-the-fly which instead is capable of streaming the dynamic scene. Our algorithm can achieve an interactive speed of 6FPS training and rendering on synthetic dynamic scenes on-the-fly, and a significant speed-up compared to the state-of-the-art on real-world dynamic scenes.
  arXiv  Detail & Related papers  (2023-05-24T07:36:47Z)
• T\"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis [32.878225196378374]
  We introduce a neural representation based on an image formation model for continuous-wave ToF cameras. We show that this approach improves robustness of dynamic scene reconstruction to erroneous calibration and large motions.
  arXiv  Detail & Related papers  (2021-09-30T17:12:59Z)
• Neural Trajectory Fields for Dynamic Novel View Synthesis [40.9665251865609]
  We introduce DCT-NeRF, a coordinatebased neural representation for dynamic scenes. We learn smooth and stable trajectories over the input sequence for each point in space. This allows us to enforce consistency between any two frames in the sequence, which results in high quality reconstruction.
  arXiv  Detail & Related papers  (2021-05-12T22:38:30Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.