DReg-NeRF: Deep Registration for Neural Radiance Fields

• URL: http://arxiv.org/abs/2308.09386v1
• Date: Fri, 18 Aug 2023 08:37:49 GMT
• Title: DReg-NeRF: Deep Registration for Neural Radiance Fields
• Authors: Yu Chen, Gim Hee Lee
• Abstract summary: We propose DReg-NeRF to solve the NeRF registration problem on object-centric annotated scenes without human intervention. Our proposed method beats the SOTA point cloud registration methods by a large margin.
• Score: 66.69049158826677
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Neural Radiance Fields (NeRF) is popular in the computer vision community recently, registering multiple NeRFs has yet to gain much attention. Unlike the existing work, NeRF2NeRF, which is based on traditional optimization methods and needs human annotated keypoints, we propose DReg-NeRF to solve the NeRF registration problem on object-centric scenes without human intervention. After training NeRF models, our DReg-NeRF first extracts features from the occupancy grid in NeRF. Subsequently, our DReg-NeRF utilizes a transformer architecture with self-attention and cross-attention layers to learn the relations between pairwise NeRF blocks. In contrast to state-of-the-art (SOTA) point cloud registration methods, the decoupled correspondences are supervised by surface fields without any ground truth overlapping labels. We construct a novel view synthesis dataset with 1,700+ 3D objects obtained from Objaverse to train our network. When evaluated on the test set, our proposed method beats the SOTA point cloud registration methods by a large margin, with a mean $\text{RPE}=9.67^{\circ}$ and a mean $\text{RTE}=0.038$. Our code is available at https://github.com/AIBluefisher/DReg-NeRF.

Related papers

• NeRFuser: Large-Scale Scene Representation by NeRF Fusion [35.749208740102546]
  A practical benefit of implicit visual representations like Neural Radiance Fields (NeRFs) is their memory efficiency. We propose NeRFuser, a novel architecture for NeRF registration and blending that assumes only access to pre-generated NeRFs.
  arXiv  Detail & Related papers  (2023-05-22T17:59:05Z)
• Learning a Diffusion Prior for NeRFs [84.99454404653339]
  We propose to use a diffusion model to generate NeRFs encoded on a regularized grid. We show that our model can sample realistic NeRFs, while at the same time allowing conditional generations, given a certain observation as guidance.
  arXiv  Detail & Related papers  (2023-04-27T19:24:21Z)
• FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization [32.1581416980828]
  We present Frequency regularized NeRF (FreeNeRF), a surprisingly simple baseline that outperforms previous methods. We analyze the key challenges in few-shot neural rendering and find that frequency plays an important role in NeRF's training.
  arXiv  Detail & Related papers  (2023-03-13T18:59:03Z)
• Removing Objects From Neural Radiance Fields [60.067117643543824]
  We propose a framework to remove objects from a NeRF representation created from an RGB-D sequence. Our NeRF inpainting method leverages recent work in 2D image inpainting and is guided by a user-provided mask. We show that our method for NeRF editing is effective for synthesizing plausible inpaintings in a multi-view coherent manner.
  arXiv  Detail & Related papers  (2022-12-22T18:51:06Z)
• NeRF-RPN: A general framework for object detection in NeRFs [54.54613914831599]
  NeRF-RPN aims to detect all bounding boxes of objects in a scene. NeRF-RPN is a general framework and can be applied to detect objects without class labels.
  arXiv  Detail & Related papers  (2022-11-21T17:02:01Z)
• NeRF-Loc: Transformer-Based Object Localization Within Neural Radiance Fields [62.89785701659139]
  We propose a transformer-based framework, NeRF-Loc, to extract 3D bounding boxes of objects in NeRF scenes. NeRF-Loc takes a pre-trained NeRF model and camera view as input and produces labeled, oriented 3D bounding boxes of objects as output.
  arXiv  Detail & Related papers  (2022-09-24T18:34:22Z)
• iNeRF: Inverting Neural Radiance Fields for Pose Estimation [68.91325516370013]
  We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural RadianceField (NeRF) NeRFs have been shown to be remarkably effective for the task of view synthesis.
  arXiv  Detail & Related papers  (2020-12-10T18:36:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.