Real-Time Scene Reconstruction using Light Field Probes

• URL: http://arxiv.org/abs/2507.14624v1
• Date: Sat, 19 Jul 2025 13:43:30 GMT
• Title: Real-Time Scene Reconstruction using Light Field Probes
• Authors: Yaru Liu, Derek Nowrouzezahri, Morgan Mcguire,
• Abstract summary: Reconstructing photo-realistic large-scale scenes from images is a long-standing problem in computer graphics.<n>Our work explores novel view synthesis methods that efficiently reconstruct complex scenes without explicit use of scene geometries.<n>Our approach can potentially be applied to virtual reality (VR) and augmented reality (AR) applications.
• Score: 2.283090308443312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing photo-realistic large-scale scenes from images, for example at city scale, is a long-standing problem in computer graphics. Neural rendering is an emerging technique that enables photo-realistic image synthesis from previously unobserved viewpoints; however, state-of-the-art neural rendering methods have difficulty efficiently rendering a high complex large-scale scene because these methods typically trade scene size, fidelity, and rendering speed for quality. The other stream of techniques utilizes scene geometries for reconstruction. But the cost of building and maintaining a large set of geometry data increases as scene size grows. Our work explores novel view synthesis methods that efficiently reconstruct complex scenes without explicit use of scene geometries. Specifically, given sparse images of the scene (captured from the real world), we reconstruct intermediate, multi-scale, implicit representations of scene geometries. In this way, our method avoids explicitly relying on scene geometry, significantly reducing the computational cost of maintaining large 3D data. Unlike current methods, we reconstruct the scene using a probe data structure. Probe data hold highly accurate depth information of dense data points, enabling the reconstruction of highly complex scenes. By reconstructing the scene using probe data, the rendering cost is independent of the complexity of the scene. As such, our approach combines geometry reconstruction and novel view synthesis. Moreover, when rendering large-scale scenes, compressing and streaming probe data is more efficient than using explicit scene geometry. Therefore, our neural representation approach can potentially be applied to virtual reality (VR) and augmented reality (AR) applications.

Related papers

• ClaraVid: A Holistic Scene Reconstruction Benchmark From Aerial Perspective With Delentropy-Based Complexity Profiling [5.6168844664788855]
  We introduce ClaraVid, a synthetic aerial dataset specifically designed to overcome limitations of existing datasets.<n>The dataset consists of 16,917 high-resolution images captured at 4032x3024 from multiple viewpoints across diverse landscapes.<n>To further advance neural reconstruction, we introduce the Delentropic Scene Profile (DSP), a novel complexity metric derived from differential entropy analysis.
  arXiv  Detail & Related papers  (2025-03-22T20:26:20Z)
• Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction [51.3632308129838]
  We present Total-Decom, a novel method for decomposed 3D reconstruction with minimal human interaction. Our approach seamlessly integrates the Segment Anything Model (SAM) with hybrid implicit-explicit neural surface representations and a mesh-based region-growing technique for accurate 3D object decomposition. We extensively evaluate our method on benchmark datasets and demonstrate its potential for downstream applications, such as animation and scene editing.
  arXiv  Detail & Related papers  (2024-03-28T11:12:33Z)
• Neural 3D Scene Reconstruction from Multiple 2D Images without 3D Supervision [41.20504333318276]
  We propose a novel neural reconstruction method that reconstructs scenes using sparse depth under the plane constraints without 3D supervision. We introduce a signed distance function field, a color field, and a probability field to represent a scene. We optimize these fields to reconstruct the scene by using differentiable ray marching with accessible 2D images as supervision.
  arXiv  Detail & Related papers  (2023-06-30T13:30:48Z)
• CompNVS: Novel View Synthesis with Scene Completion [83.19663671794596]
  We propose a generative pipeline performing on a sparse grid-based neural scene representation to complete unobserved scene parts. We process encoded image features in 3D space with a geometry completion network and a subsequent texture inpainting network to extrapolate the missing area. Photorealistic image sequences can be finally obtained via consistency-relevant differentiable rendering.
  arXiv  Detail & Related papers  (2022-07-23T09:03:13Z)
• PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes [84.66946637534089]
  PhotoScene is a framework that takes input image(s) of a scene and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes.
  arXiv  Detail & Related papers  (2022-07-02T06:52:44Z)
• ERF: Explicit Radiance Field Reconstruction From Scratch [12.254150867994163]
  We propose a novel explicit dense 3D reconstruction approach that processes a set of images of a scene with sensor poses and calibrations and estimates a photo-real digital model. One of the key innovations is that the underlying volumetric representation is completely explicit. We show that our method is general and practical. It does not require a highly controlled lab setup for capturing, but allows for reconstructing scenes with a vast variety of objects.
  arXiv  Detail & Related papers  (2022-02-28T19:37:12Z)
• Advances in Neural Rendering [115.05042097988768]
  This report focuses on methods that combine classical rendering with learned 3D scene representations. A key advantage of these methods is that they are 3D-consistent by design, enabling applications such as novel viewpoint of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling non-rigidly deforming objects.
  arXiv  Detail & Related papers  (2021-11-10T18:57:01Z)
• RetrievalFuse: Neural 3D Scene Reconstruction with a Database [34.44425679892233]
  We introduce a new method that directly leverages scene geometry from the training database. First, we learn to synthesize an initial estimate for a 3D scene, constructed by retrieving a top-k set of volumetric chunks from the scene database. These candidates are then refined to a final scene generation with an attention-based refinement that can effectively select the most consistent set of geometry from the candidates. We demonstrate our neural scene reconstruction with a database for the tasks of 3D super resolution and surface reconstruction from sparse point clouds.
  arXiv  Detail & Related papers  (2021-03-31T18:00:09Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)
• Neural Sparse Voxel Fields [151.20366604586403]
  We introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving higher quality results.
  arXiv  Detail & Related papers  (2020-07-22T17:51:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.