iNeRF: Inverting Neural Radiance Fields for Pose Estimation

• URL: http://arxiv.org/abs/2012.05877v2
• Date: Thu, 1 Apr 2021 18:51:40 GMT
• Title: iNeRF: Inverting Neural Radiance Fields for Pose Estimation
• Authors: Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, Tsung-Yi Lin
• Abstract summary: We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural RadianceField (NeRF) NeRFs have been shown to be remarkably effective for the task of view synthesis.
• Score: 68.91325516370013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural RadianceField (NeRF). NeRFs have been shown to be remarkably effective for the task of view synthesis - synthesizing photorealistic novel views of real-world scenes or objects. In this work, we investigate whether we can apply analysis-by-synthesis via NeRF for mesh-free, RGB-only 6DoF pose estimation - given an image, find the translation and rotation of a camera relative to a 3D object or scene. Our method assumes that no object mesh models are available during either training or test time. Starting from an initial pose estimate, we use gradient descent to minimize the residual between pixels rendered from a NeRF and pixels in an observed image. In our experiments, we first study 1) how to sample rays during pose refinement for iNeRF to collect informative gradients and 2) how different batch sizes of rays affect iNeRF on a synthetic dataset. We then show that for complex real-world scenes from the LLFF dataset, iNeRF can improve NeRF by estimating the camera poses of novel images and using these images as additional training data for NeRF. Finally, we show iNeRF can perform category-level object pose estimation, including object instances not seen during training, with RGB images by inverting a NeRF model inferred from a single view.

Related papers

• NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
  Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. We address these issues with an approach based on ray tracing. Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
  arXiv  Detail & Related papers  (2024-05-23T17:59:57Z)
• IFFNeRF: Initialisation Free and Fast 6DoF pose estimation from a single image and a NeRF model [17.202161109957608]
  We introduce IFFNeRF to estimate the six degrees-of-freedom (6DoF) camera pose of a given image. IFFNeRF is specifically designed to operate in real-time and eliminates the need for an initial pose guess.
  arXiv  Detail & Related papers  (2024-03-19T12:36:51Z)
• CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning [23.080474939586654]
  We propose a novel underlinecamera parameter underlinefree neural radiance field (CF-NeRF) CF-NeRF incrementally reconstructs 3D representations and recovers the camera parameters inspired by incremental structure from motion. Results demonstrate that CF-NeRF is robust to camera rotation and achieves state-of-the-art results without providing prior information and constraints.
  arXiv  Detail & Related papers  (2023-12-14T09:09:31Z)
• IL-NeRF: Incremental Learning for Neural Radiance Fields with Camera Pose Alignment [12.580584725107173]
  We propose IL-NeRF, a novel framework for incremental NeRF training. We show that IL-NeRF handles incremental NeRF training and outperforms the baselines by up to $54.04%$ in rendering quality.
  arXiv  Detail & Related papers  (2023-12-10T04:12:27Z)
• NeRFuser: Large-Scale Scene Representation by NeRF Fusion [35.749208740102546]
  A practical benefit of implicit visual representations like Neural Radiance Fields (NeRFs) is their memory efficiency. We propose NeRFuser, a novel architecture for NeRF registration and blending that assumes only access to pre-generated NeRFs.
  arXiv  Detail & Related papers  (2023-05-22T17:59:05Z)
• NeRF applied to satellite imagery for surface reconstruction [5.027411102165872]
  We present Surf-NeRF, a modified implementation of the recently introduced Shadow Neural Radiance Field (S-NeRF) model. This method is able to synthesize novel views from a sparse set of satellite images of a scene, while accounting for the variation in lighting present in the pictures. The trained model can also be used to accurately estimate the surface elevation of the scene, which is often a desirable quantity for satellite observation applications.
  arXiv  Detail & Related papers  (2023-04-09T01:37:13Z)
• StegaNeRF: Embedding Invisible Information within Neural Radiance Fields [61.653702733061785]
  We present StegaNeRF, a method for steganographic information embedding in NeRF renderings. We design an optimization framework allowing accurate hidden information extractions from images rendered by NeRF. StegaNeRF signifies an initial exploration into the novel problem of instilling customizable, imperceptible, and recoverable information to NeRF renderings.
  arXiv  Detail & Related papers  (2022-12-03T12:14:19Z)
• Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations [111.08941206369508]
  We propose Augmented NeRF (Aug-NeRF), which for the first time brings the power of robust data augmentations into regularizing the NeRF training. Our proposal learns to seamlessly blend worst-case perturbations into three distinct levels of the NeRF pipeline. Aug-NeRF effectively boosts NeRF performance in both novel view synthesis and underlying geometry reconstruction.
  arXiv  Detail & Related papers  (2022-07-04T02:27:07Z)
• R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis [76.07010495581535]
  Rendering a single pixel requires querying the Neural Radiance Field network hundreds of times. NeLF presents a more straightforward representation over NeRF in novel view. We show the key to successfully learning a deep NeLF network is to have sufficient data.
  arXiv  Detail & Related papers  (2022-03-31T17:57:05Z)
• NeRF-SR: High-Quality Neural Radiance Fields using Super-Sampling [82.99453001445478]
  We present NeRF-SR, a solution for high-resolution (HR) novel view synthesis with mostly low-resolution (LR) inputs. Our method is built upon Neural Radiance Fields (NeRF) that predicts per-point density and color with a multi-layer perceptron.
  arXiv  Detail & Related papers  (2021-12-03T07:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.