Mask-Based Modeling for Neural Radiance Fields

• URL: http://arxiv.org/abs/2304.04962v2
• Date: Tue, 19 Mar 2024 08:28:46 GMT
• Title: Mask-Based Modeling for Neural Radiance Fields
• Authors: Ganlin Yang, Guoqiang Wei, Zhizheng Zhang, Yan Lu, Dong Liu,
• Abstract summary: In this work, we unveil that 3D implicit representation learning can be significantly improved by mask-based modeling. We propose MRVM-NeRF, which is a self-supervised pretraining target to predict complete scene representations from partially masked features along each ray. With this pretraining target, MRVM-NeRF enables better use of correlations across different points and views as the geometry priors.
• Score: 20.728248301818912
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most Neural Radiance Fields (NeRFs) exhibit limited generalization capabilities, which restrict their applicability in representing multiple scenes using a single model. To address this problem, existing generalizable NeRF methods simply condition the model on image features. These methods still struggle to learn precise global representations over diverse scenes since they lack an effective mechanism for interacting among different points and views. In this work, we unveil that 3D implicit representation learning can be significantly improved by mask-based modeling. Specifically, we propose masked ray and view modeling for generalizable NeRF (MRVM-NeRF), which is a self-supervised pretraining target to predict complete scene representations from partially masked features along each ray. With this pretraining target, MRVM-NeRF enables better use of correlations across different points and views as the geometry priors, which thereby strengthens the capability of capturing intricate details within the scenes and boosts the generalization capability across different scenes. Extensive experiments demonstrate the effectiveness of our proposed MRVM-NeRF on both synthetic and real-world datasets, qualitatively and quantitatively. Besides, we also conduct experiments to show the compatibility of our proposed method with various backbones and its superiority under few-shot cases.

Related papers

• Inter-slice Super-resolution of Magnetic Resonance Images by Pre-training and Self-supervised Fine-tuning [49.197385954021456]
  In clinical practice, 2D magnetic resonance (MR) sequences are widely adopted. While individual 2D slices can be stacked to form a 3D volume, the relatively large slice spacing can pose challenges for visualization and subsequent analysis tasks. To reduce slice spacing, deep-learning-based super-resolution techniques are widely investigated. Most current solutions require a substantial number of paired high-resolution and low-resolution images for supervised training, which are typically unavailable in real-world scenarios.
  arXiv  Detail & Related papers  (2024-06-10T02:20:26Z)
• Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling [70.34875558830241]
  We present a way for learning a-temporal (4D) embedding, based on semantic semantic gears to allow for stratified modeling of dynamic regions of rendering the scene. At the same time, almost for free, our tracking approach enables free-viewpoint of interest - a functionality not yet achieved by existing NeRF-based methods.
  arXiv  Detail & Related papers  (2024-06-06T03:37:39Z)
• Divide and Conquer: Rethinking the Training Paradigm of Neural Radiance Fields [24.99410489251996]
  Neural radiance fields (NeRFs) have exhibited potential in high-fidelity views of 3D scenes. Standard training paradigm of NeRF presupposes an equal importance for each image in the training set. In this paper, we take a closer look at the implications of the current training paradigm and redesign this for more superior rendering quality.
  arXiv  Detail & Related papers  (2024-01-29T13:23:34Z)
• 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands [51.305421495638434]
  Neural radiance fields (NeRFs) are promising 3D representations for scenes, objects, and humans. This paper proposes a generalizable visibility-aware NeRF framework for interacting hands. Experiments on the Interhand2.6M dataset demonstrate that our proposed VA-NeRF outperforms conventional NeRFs significantly.
  arXiv  Detail & Related papers  (2024-01-02T00:42:06Z)
• Towards a Robust Framework for NeRF Evaluation [11.348562090906576]
  We propose a new test framework which isolates the neural rendering network from the Neural Radiance Field (NeRF) pipeline. We then perform a parametric evaluation by training and evaluating the NeRF on an explicit radiance field representation. Our approach offers the potential to create a comparative objective evaluation framework for NeRF methods.
  arXiv  Detail & Related papers  (2023-05-29T13:30:26Z)
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction [77.69363640021503]
  3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. We present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects.
  arXiv  Detail & Related papers  (2023-04-13T17:59:01Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• NeRF, meet differential geometry! [10.269997499911668]
  We show how differential geometry can provide regularization tools for robustly training NeRF-like models. We show how these tools yield a direct mathematical formalism of previously proposed NeRF variants aimed at improving the performance in challenging conditions.
  arXiv  Detail & Related papers  (2022-06-29T22:45:34Z)
• UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction [61.17219252031391]
  We present a novel method for reconstructing surfaces from multi-view images using Neural implicit 3D representations. Our key insight is that implicit surface models and radiance fields can be formulated in a unified way, enabling both surface and volume rendering. Our experiments demonstrate that we outperform NeRF in terms of reconstruction quality while performing on par with IDR without requiring masks.
  arXiv  Detail & Related papers  (2021-04-20T15:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.