BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis

• URL: http://arxiv.org/abs/2302.14859v2
• Date: Tue, 16 May 2023 15:01:42 GMT
• Title: BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis
• Authors: Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, Ben Mildenhall
• Abstract summary: We present a method for reconstructing high-quality meshes of large real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians.
• Score: 42.93055827628597
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation.

Related papers

• LinPrim: Linear Primitives for Differentiable Volumetric Rendering [53.780682194322225]
  We introduce two new scene representations based on linear primitives-octahedra and tetrahedra-both of which define homogeneous volumes bounded by triangular faces. This formulation aligns naturally with standard mesh-based tools, minimizing overhead for downstream applications. We demonstrate comparable performance to state-of-the-art volumetric methods while requiring fewer primitives to achieve similar reconstruction fidelity.
  arXiv  Detail & Related papers  (2025-01-27T18:49:38Z)
• EnvGS: Modeling View-Dependent Appearance with Environment Gaussian [78.74634059559891]
  EnvGS is a novel approach that employs a set of Gaussian primitives as an explicit 3D representation for capturing reflections of environments. To efficiently render these environment Gaussian primitives, we developed a ray-tracing-based reflection that leverages the GPU's RT core for fast rendering. Results from multiple real-world and synthetic datasets demonstrate that our method produces significantly more detailed reflections.
  arXiv  Detail & Related papers  (2024-12-19T18:59:57Z)
• D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
  This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures. Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
  arXiv  Detail & Related papers  (2024-06-14T14:35:44Z)
• RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS [47.47003067842151]
  We present RadSplat, a lightweight method for robust real-time rendering of complex scenes. First, we use radiance fields as a prior and supervision signal for optimizing point-based scene representations, leading to improved quality and more robust optimization. Next, we develop a novel pruning technique reducing the overall point count while maintaining high quality, leading to smaller and more compact scene representations with faster inference speeds.
  arXiv  Detail & Related papers  (2024-03-20T17:59:55Z)
• VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction [59.40711222096875]
  We present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets.
  arXiv  Detail & Related papers  (2024-02-27T11:40:50Z)
• High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering [15.009484906668737]
  We introduce a novel technique that reconstructs mesh-based blendshape rigs from single or sparse multi-view videos. Experiments demonstrate that, with the flexible input of single or sparse multi-view videos, we reconstruct personalized high-fidelity blendshapes.
  arXiv  Detail & Related papers  (2024-01-16T14:41:31Z)
• HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces [71.1071688018433]
  Neural radiance fields provide state-of-the-art view synthesis quality but tend to be slow to render. We propose a method, HybridNeRF, that leverages the strengths of both representations by rendering most objects as surfaces. We improve error rates by 15-30% while achieving real-time framerates (at least 36 FPS) for virtual-reality resolutions (2Kx2K)
  arXiv  Detail & Related papers  (2023-12-05T22:04:49Z)
• Neural Lumigraph Rendering [33.676795978166375]
  State-of-the-art (SOTA) neural volume rendering approaches are slow to train and require minutes of inference (i.e., rendering) time for high image resolutions. We adopt high-capacity neural scene representations with periodic activations for jointly optimizing an implicit surface and a radiance field of a scene supervised exclusively with posed 2D images. Our neural rendering pipeline accelerates SOTA neural volume rendering by about two orders of magnitude and our implicit surface representation is unique in allowing us to export a mesh with view-dependent texture information.
  arXiv  Detail & Related papers  (2021-03-22T03:46:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.