NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis

• URL: http://arxiv.org/abs/2312.13328v2
• Date: Mon, 22 Apr 2024 15:05:18 GMT
• Title: NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis
• Authors: Zinuo You, Andreas Geiger, Anpei Chen,
• Abstract summary: NeLF-Pro is a novel representation to model and reconstruct light fields in diverse natural scenes. Our central idea is to bake the scene's light field into spatially varying learnable representations.
• Score: 27.362216326282145
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present NeLF-Pro, a novel representation to model and reconstruct light fields in diverse natural scenes that vary in extent and spatial granularity. In contrast to previous fast reconstruction methods that represent the 3D scene globally, we model the light field of a scene as a set of local light field feature probes, parameterized with position and multi-channel 2D feature maps. Our central idea is to bake the scene's light field into spatially varying learnable representations and to query point features by weighted blending of probes close to the camera - allowing for mipmap representation and rendering. We introduce a novel vector-matrix-matrix (VMM) factorization technique that effectively represents the light field feature probes as products of core factors (i.e., VM) shared among local feature probes, and a basis factor (i.e., M) - efficiently encoding internal relationships and patterns within the scene. Experimentally, we demonstrate that NeLF-Pro significantly boosts the performance of feature grid-based representations, and achieves fast reconstruction with better rendering quality while maintaining compact modeling. Project webpage https://sinoyou.github.io/nelf-pro/.

Related papers

• PBIR-NIE: Glossy Object Capture under Non-Distant Lighting [30.325872237020395]
  Glossy objects present a significant challenge for 3D reconstruction from multi-view input images under natural lighting. We introduce PBIR-NIE, an inverse rendering framework designed to holistically capture the geometry, material attributes, and surrounding illumination of such objects.
  arXiv  Detail & Related papers  (2024-08-13T13:26:24Z)
• Depth Reconstruction with Neural Signed Distance Fields in Structured Light Systems [15.603880588503355]
  We introduce a novel depth estimation technique for multi-frame structured light setups using neural implicit representations of 3D space. Our approach employs a neural signed distance field (SDF), trained through self-supervised differentiable rendering.
  arXiv  Detail & Related papers  (2024-05-20T13:24:35Z)
• MMPI: a Flexible Radiance Field Representation by Multiple Multi-plane Images Blending [61.45757368117578]
  This paper presents a flexible representation of neural radiance fields based on multi-plane images (MPI) MPI is widely used in NeRF learning for its simple definition, easy calculation, and powerful ability to represent unbounded scenes. We show that MPI can synthesize high-quality novel views of complex scenes with diverse camera distributions and view directions.
  arXiv  Detail & Related papers  (2023-09-30T04:36:43Z)
• Neural Projection Mapping Using Reflectance Fields [11.74757574153076]
  We introduce a projector into a neural reflectance field that allows to calibrate the projector and photo realistic light editing. Our neural field consists of three neural networks, estimating geometry, material, and transmittance. We believe that neural projection mapping opens up the door to novel and exciting downstream tasks, through the joint optimization of the scene and projection images.
  arXiv  Detail & Related papers  (2023-06-11T05:33:10Z)
• Multi-Plane Neural Radiance Fields for Novel View Synthesis [5.478764356647437]
  Novel view synthesis is a long-standing problem that revolves around rendering frames of scenes from novel camera viewpoints. In this work, we examine the performance, generalization, and efficiency of single-view multi-plane neural radiance fields. We propose a new multiplane NeRF architecture that accepts multiple views to improve the synthesis results and expand the viewing range.
  arXiv  Detail & Related papers  (2023-03-03T06:32:55Z)
• Neural Point Light Fields [80.98651520818785]
  We introduce Neural Point Light Fields that represent scenes implicitly with a light field living on a sparse point cloud. These point light fields are as a function of the ray direction, and local point feature neighborhood, allowing us to interpolate the light field conditioned training images without dense object coverage and parallax.
  arXiv  Detail & Related papers  (2021-12-02T18:20:10Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering [60.02806355570514]
  Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. We propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field. Rendering a ray from an LFN requires only a *single* network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or based on volumetrics.
  arXiv  Detail & Related papers  (2021-06-04T17:54:49Z)
• Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications [78.63280020581662]
  A novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-03-24T08:16:32Z)
• Crowdsampling the Plenoptic Function [56.10020793913216]
  We present a new approach to novel view synthesis under time-varying illumination from such data. We introduce a new DeepMPI representation, motivated by observations on the sparsity structure of the plenoptic function. Our method can synthesize the same compelling parallax and view-dependent effects as previous MPI methods, while simultaneously interpolating along changes in reflectance and illumination with time.
  arXiv  Detail & Related papers  (2020-07-30T02:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.