GARF:Geometry-Aware Generalized Neural Radiance Field

• URL: http://arxiv.org/abs/2212.02280v2
• Date: Wed, 7 Dec 2022 07:52:06 GMT
• Title: GARF:Geometry-Aware Generalized Neural Radiance Field
• Authors: Yue Shi, Dingyi Rong, Bingbing Ni, Chang Chen, Wenjun Zhang
• Abstract summary: We propose Geometry-Aware Generalized Neural Radiance Field (GARF) with a geometry-aware dynamic sampling (GADS) strategy. Our framework infers the unseen scenes on both pixel-scale and geometry-scale with only a few input images. Experiments on indoor and outdoor datasets show that GARF reduces samples by more than 25%, while improving rendering quality and 3D geometry estimation.
• Score: 47.76524984421343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Field (NeRF) has revolutionized free viewpoint rendering tasks and achieved impressive results. However, the efficiency and accuracy problems hinder its wide applications. To address these issues, we propose Geometry-Aware Generalized Neural Radiance Field (GARF) with a geometry-aware dynamic sampling (GADS) strategy to perform real-time novel view rendering and unsupervised depth estimation on unseen scenes without per-scene optimization. Distinct from most existing generalized NeRFs, our framework infers the unseen scenes on both pixel-scale and geometry-scale with only a few input images. More specifically, our method learns common attributes of novel-view synthesis by an encoder-decoder structure and a point-level learnable multi-view feature fusion module which helps avoid occlusion. To preserve scene characteristics in the generalized model, we introduce an unsupervised depth estimation module to derive the coarse geometry, narrow down the ray sampling interval to proximity space of the estimated surface and sample in expectation maximum position, constituting Geometry-Aware Dynamic Sampling strategy (GADS). Moreover, we introduce a Multi-level Semantic Consistency loss (MSC) to assist more informative representation learning. Extensive experiments on indoor and outdoor datasets show that comparing with state-of-the-art generalized NeRF methods, GARF reduces samples by more than 25\%, while improving rendering quality and 3D geometry estimation.

Related papers

• Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation [7.203073346844801]
  This paper introduces a novel paradigm for the generalizable neural radiance field (NeRF) We propose the first paradigm that constructs the generalizable neural field based on point-based rather than image-based rendering. Our approach explicitly models visibilities by geometric priors and augments them with neural features.
  arXiv  Detail & Related papers  (2024-01-25T17:58:51Z)
• HG3-NeRF: Hierarchical Geometric, Semantic, and Photometric Guided Neural Radiance Fields for Sparse View Inputs [7.715395970689711]
  We introduce Hierarchical Geometric, Semantic, and Photometric Guided NeRF (HG3-NeRF) HG3-NeRF is a novel methodology that can address the limitation and enhance consistency of geometry, semantic content, and appearance across different views. Experimental results demonstrate that HG3-NeRF can outperform other state-of-the-art methods on different standard benchmarks.
  arXiv  Detail & Related papers  (2024-01-22T06:28:08Z)
• NeRF-VO: Real-Time Sparse Visual Odometry with Neural Radiance Fields [13.178099653374945]
  NeRF-VO integrates learning-based sparse visual odometry for low-latency camera tracking and a neural radiance scene representation. We surpass SOTA methods in pose estimation accuracy, novel view fidelity, and dense reconstruction quality across a variety of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-12-20T22:42:17Z)
• Anti-Aliased Neural Implicit Surfaces with Encoding Level of Detail [54.03399077258403]
  We present LoD-NeuS, an efficient neural representation for high-frequency geometry detail recovery and anti-aliased novel view rendering. Our representation aggregates space features from a multi-convolved featurization within a conical frustum along a ray.
  arXiv  Detail & Related papers  (2023-09-19T05:44:00Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• Learning Neural Radiance Fields from Multi-View Geometry [1.1011268090482573]
  We present a framework, called MVG-NeRF, that combines Multi-View Geometry algorithms and Neural Radiance Fields (NeRF) for image-based 3D reconstruction. NeRF has revolutionized the field of implicit 3D representations, mainly due to a differentiable rendering formulation that enables high-quality and geometry-aware novel view synthesis.
  arXiv  Detail & Related papers  (2022-10-24T08:53:35Z)
• CLONeR: Camera-Lidar Fusion for Occupancy Grid-aided Neural Representations [77.90883737693325]
  This paper proposes CLONeR, which significantly improves upon NeRF by allowing it to model large outdoor driving scenes observed from sparse input sensor views. This is achieved by decoupling occupancy and color learning within the NeRF framework into separate Multi-Layer Perceptrons (MLPs) trained using LiDAR and camera data, respectively. In addition, this paper proposes a novel method to build differentiable 3D Occupancy Grid Maps (OGM) alongside the NeRF model, and leverage this occupancy grid for improved sampling of points along a ray for rendering in metric space.
  arXiv  Detail & Related papers  (2022-09-02T17:44:50Z)
• PVSeRF: Joint Pixel-, Voxel- and Surface-Aligned Radiance Field for Single-Image Novel View Synthesis [52.546998369121354]
  We present PVSeRF, a learning framework that reconstructs neural radiance fields from single-view RGB images. We propose to incorporate explicit geometry reasoning and combine it with pixel-aligned features for radiance field prediction. We show that the introduction of such geometry-aware features helps to achieve a better disentanglement between appearance and geometry.
  arXiv  Detail & Related papers  (2022-02-10T07:39:47Z)
• Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering [139.159534903657]
  We develop a generalizable and efficient Neural Radiance Field (NeRF) pipeline for high-fidelity free-viewpoint human body details. To better tackle self-occlusion, we devise a geometry-guided multi-view feature integration approach. For achieving higher rendering efficiency, we introduce a geometry-guided progressive rendering pipeline.
  arXiv  Detail & Related papers  (2021-12-08T14:42:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.