HG3-NeRF: Hierarchical Geometric, Semantic, and Photometric Guided Neural Radiance Fields for Sparse View Inputs

• URL: http://arxiv.org/abs/2401.11711v1
• Date: Mon, 22 Jan 2024 06:28:08 GMT
• Title: HG3-NeRF: Hierarchical Geometric, Semantic, and Photometric Guided Neural Radiance Fields for Sparse View Inputs
• Authors: Zelin Gao, Weichen Dai, Yu Zhang
• Abstract summary: We introduce Hierarchical Geometric, Semantic, and Photometric Guided NeRF (HG3-NeRF) HG3-NeRF is a novel methodology that can address the limitation and enhance consistency of geometry, semantic content, and appearance across different views. Experimental results demonstrate that HG3-NeRF can outperform other state-of-the-art methods on different standard benchmarks.
• Score: 7.715395970689711
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) have garnered considerable attention as a paradigm for novel view synthesis by learning scene representations from discrete observations. Nevertheless, NeRF exhibit pronounced performance degradation when confronted with sparse view inputs, consequently curtailing its further applicability. In this work, we introduce Hierarchical Geometric, Semantic, and Photometric Guided NeRF (HG3-NeRF), a novel methodology that can address the aforementioned limitation and enhance consistency of geometry, semantic content, and appearance across different views. We propose Hierarchical Geometric Guidance (HGG) to incorporate the attachment of Structure from Motion (SfM), namely sparse depth prior, into the scene representations. Different from direct depth supervision, HGG samples volume points from local-to-global geometric regions, mitigating the misalignment caused by inherent bias in the depth prior. Furthermore, we draw inspiration from notable variations in semantic consistency observed across images of different resolutions and propose Hierarchical Semantic Guidance (HSG) to learn the coarse-to-fine semantic content, which corresponds to the coarse-to-fine scene representations. Experimental results demonstrate that HG3-NeRF can outperform other state-of-the-art methods on different standard benchmarks and achieve high-fidelity synthesis results for sparse view inputs.

Related papers

• Efficient Depth-Guided Urban View Synthesis [52.841803876653465]
  We introduce Efficient Depth-Guided Urban View Synthesis (EDUS) for fast feed-forward inference and efficient per-scene fine-tuning. EDUS exploits noisy predicted geometric priors as guidance to enable generalizable urban view synthesis from sparse input images. Our results indicate that EDUS achieves state-of-the-art performance in sparse view settings when combined with fast test-time optimization.
  arXiv  Detail & Related papers  (2024-07-17T08:16:25Z)
• GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding [30.951440204237166]
  We introduce a Generalizable Semantic Neural Radiance Field (GSNeRF), which takes image semantics into the synthesis process. Our GSNeRF is composed of two stages: Semantic Geo-Reasoning and Depth-Guided Visual rendering.
  arXiv  Detail & Related papers  (2024-03-06T10:55:50Z)
• Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation [7.203073346844801]
  This paper introduces a novel paradigm for the generalizable neural radiance field (NeRF) We propose the first paradigm that constructs the generalizable neural field based on point-based rather than image-based rendering. Our approach explicitly models visibilities by geometric priors and augments them with neural features.
  arXiv  Detail & Related papers  (2024-01-25T17:58:51Z)
• GS-IR: 3D Gaussian Splatting for Inverse Rendering [71.14234327414086]
  We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) We extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering.
  arXiv  Detail & Related papers  (2023-11-26T02:35:09Z)
• GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding [101.32590239809113]
  Generalized Perception NeRF (GP-NeRF) is a novel pipeline that makes the widely used segmentation model and NeRF work compatibly under a unified framework. We propose two self-distillation mechanisms, i.e., the Semantic Distill Loss and the Depth-Guided Semantic Distill Loss, to enhance the discrimination and quality of the semantic field.
  arXiv  Detail & Related papers  (2023-11-20T15:59:41Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates [16.344734292989504]
  SCADE is a novel technique that improves NeRF reconstruction quality on sparse, unconstrained input views. We propose a new method that learns to predict, for each view, a continuous, multimodal distribution of depth estimates. Experiments show that our approach enables higher fidelity novel view synthesis from sparse views.
  arXiv  Detail & Related papers  (2023-03-23T18:00:07Z)
• GARF:Geometry-Aware Generalized Neural Radiance Field [47.76524984421343]
  We propose Geometry-Aware Generalized Neural Radiance Field (GARF) with a geometry-aware dynamic sampling (GADS) strategy. Our framework infers the unseen scenes on both pixel-scale and geometry-scale with only a few input images. Experiments on indoor and outdoor datasets show that GARF reduces samples by more than 25%, while improving rendering quality and 3D geometry estimation.
  arXiv  Detail & Related papers  (2022-12-05T14:00:59Z)
• 3D Dense Geometry-Guided Facial Expression Synthesis by Adversarial Learning [54.24887282693925]
  We propose a novel framework to exploit 3D dense (depth and surface normals) information for expression manipulation. We use an off-the-shelf state-of-the-art 3D reconstruction model to estimate the depth and create a large-scale RGB-Depth dataset. Our experiments demonstrate that the proposed method outperforms the competitive baseline and existing arts by a large margin.
  arXiv  Detail & Related papers  (2020-09-30T17:12:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.