Semantic-aware Occlusion Filtering Neural Radiance Fields in the Wild

• URL: http://arxiv.org/abs/2303.03966v1
• Date: Sun, 5 Mar 2023 11:50:34 GMT
• Title: Semantic-aware Occlusion Filtering Neural Radiance Fields in the Wild
• Authors: Jaewon Lee, Injae Kim, Hwan Heo, Hyunwoo J. Kim
• Abstract summary: We present a learning framework for reconstructing neural scene representations from unconstrained tourist photos. We introduce SF-NeRF, aiming to disentangle the static and transient components with only a few images given. We present two techniques to prevent ambiguous decomposition and noisy results of the filtering module.
• Score: 10.066261691282016
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a learning framework for reconstructing neural scene representations from a small number of unconstrained tourist photos. Since each image contains transient occluders, decomposing the static and transient components is necessary to construct radiance fields with such in-the-wild photographs where existing methods require a lot of training data. We introduce SF-NeRF, aiming to disentangle those two components with only a few images given, which exploits semantic information without any supervision. The proposed method contains an occlusion filtering module that predicts the transient color and its opacity for each pixel, which enables the NeRF model to solely learn the static scene representation. This filtering module learns the transient phenomena guided by pixel-wise semantic features obtained by a trainable image encoder that can be trained across multiple scenes to learn the prior of transient objects. Furthermore, we present two techniques to prevent ambiguous decomposition and noisy results of the filtering module. We demonstrate that our method outperforms state-of-the-art novel view synthesis methods on Phototourism dataset in a few-shot setting.

Related papers

• MLI-NeRF: Multi-Light Intrinsic-Aware Neural Radiance Fields [21.057216351934688]
  Current methods for extracting intrinsic image components, such as reflectance and shading, rely on statistical priors. We propose MLI-NeRF, which integrates textbfMultiple textbfLight information in textbfIntrinsic-aware textbfNeural textbfRadiance textbfFields.
  arXiv  Detail & Related papers  (2024-11-26T08:57:38Z)
• Sparse-DeRF: Deblurred Neural Radiance Fields from Sparse View [17.214047499850487]
  This paper focuses on constructing deblurred neural radiance fields (DeRF) from sparse-view for more pragmatic real-world scenarios. Sparse-DeRF successfully regularizes the complicated joint optimization, presenting alleviated overfitting artifacts and enhanced quality on radiance fields. We demonstrate the effectiveness of the Sparse-DeRF with extensive quantitative and qualitative experimental results by training DeRF from 2-view, 4-view, and 6-view blurry images.
  arXiv  Detail & Related papers  (2024-07-09T07:36:54Z)
• Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling [70.34875558830241]
  We present a way for learning a-temporal (4D) embedding, based on semantic semantic gears to allow for stratified modeling of dynamic regions of rendering the scene. At the same time, almost for free, our tracking approach enables free-viewpoint of interest - a functionality not yet achieved by existing NeRF-based methods.
  arXiv  Detail & Related papers  (2024-06-06T03:37:39Z)
• Neural Image Re-Exposure [86.42475408644822]
  An improper shutter may lead to a blurry image, video discontinuity, or rolling shutter artifact. We propose a neural network-based image re-exposure framework. It consists of an encoder for visual latent space construction, a re-exposure module for aggregating information to neural film with a desired shutter strategy, and a decoder for 'developing' neural film into a desired image.
  arXiv  Detail & Related papers  (2023-05-23T01:55:37Z)
• Neural Congealing: Aligning Images to a Joint Semantic Atlas [14.348512536556413]
  We present a zero-shot self-supervised framework for aligning semantically-common content across a set of images. Our approach harnesses the power of pre-trained DINO-ViT features to learn. We show that our method performs favorably compared to a state-of-the-art method that requires extensive training on large-scale datasets.
  arXiv  Detail & Related papers  (2023-02-08T09:26:22Z)
• im2nerf: Image to Neural Radiance Field in the Wild [47.18702901448768]
  im2nerf is a learning framework that predicts a continuous neural object representation given a single input image in the wild. We show that im2nerf achieves the state-of-the-art performance for novel view synthesis from a single-view unposed image in the wild.
  arXiv  Detail & Related papers  (2022-09-08T23:28:56Z)
• Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid [102.24539566851809]
  Restoring reasonable and realistic content for arbitrary missing regions in images is an important yet challenging task. Recent image inpainting models have made significant progress in generating vivid visual details, but they can still lead to texture blurring or structural distortions. We propose the Semantic Pyramid Network (SPN) motivated by the idea that learning multi-scale semantic priors can greatly benefit the recovery of locally missing content in images.
  arXiv  Detail & Related papers  (2021-12-08T04:33:33Z)
• Recursive Self-Improvement for Camera Image and Signal Processing Pipeline [6.318974730864278]
  Current camera image and signal processing pipelines (ISPs) tend to apply a single filter that is uniformly applied to the entire image. This despite the fact that most acquired camera images have spatially heterogeneous artifacts. We present a deep reinforcement learning model that works in learned latent subspaces.
  arXiv  Detail & Related papers  (2021-11-15T02:23:40Z)
• iNeRF: Inverting Neural Radiance Fields for Pose Estimation [68.91325516370013]
  We present iNeRF, a framework that performs mesh-free pose estimation by "inverting" a Neural RadianceField (NeRF) NeRFs have been shown to be remarkably effective for the task of view synthesis.
  arXiv  Detail & Related papers  (2020-12-10T18:36:40Z)
• Self-Adaptively Learning to Demoire from Focused and Defocused Image Pairs [97.67638106818613]
  Moire artifacts are common in digital photography, resulting from the interference between high-frequency scene content and the color filter array of the camera. Existing deep learning-based demoireing methods trained on large scale iteration are limited in handling various complex moire patterns. We propose a self-adaptive learning method for demoireing a high-frequency image, with the help of an additional defocused moire-free blur image.
  arXiv  Detail & Related papers  (2020-11-03T23:09:02Z)
• NeRF++: Analyzing and Improving Neural Radiance Fields [117.73411181186088]
  Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings. NeRF fits multi-layer perceptrons representing view-invariant opacity and view-dependent color volumes to a set of training images. We address a parametrization issue involved in applying NeRF to 360 captures of objects within large-scale, 3D scenes.
  arXiv  Detail & Related papers  (2020-10-15T03:24:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.