Dynamic Multi-View Scene Reconstruction Using Neural Implicit Surface

• URL: http://arxiv.org/abs/2303.00050v1
• Date: Tue, 28 Feb 2023 19:47:30 GMT
• Title: Dynamic Multi-View Scene Reconstruction Using Neural Implicit Surface
• Authors: Decai Chen, Haofei Lu, Ingo Feldmann, Oliver Schreer, Peter Eisert
• Abstract summary: We propose a template-free method to reconstruct surface geometry and appearance using neural implicit representations from multi-view videos. We leverage topology-aware deformation and the signed distance field to learn complex dynamic surfaces via differentiable volume rendering. Experiments on different multi-view video datasets demonstrate that our method achieves high-fidelity surface reconstruction as well as photorealistic novel view synthesis.
• Score: 0.9134661726886928
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reconstructing general dynamic scenes is important for many computer vision and graphics applications. Recent works represent the dynamic scene with neural radiance fields for photorealistic view synthesis, while their surface geometry is under-constrained and noisy. Other works introduce surface constraints to the implicit neural representation to disentangle the ambiguity of geometry and appearance field for static scene reconstruction. To bridge the gap between rendering dynamic scenes and recovering static surface geometry, we propose a template-free method to reconstruct surface geometry and appearance using neural implicit representations from multi-view videos. We leverage topology-aware deformation and the signed distance field to learn complex dynamic surfaces via differentiable volume rendering without scene-specific prior knowledge like template models. Furthermore, we propose a novel mask-based ray selection strategy to significantly boost the optimization on challenging time-varying regions. Experiments on different multi-view video datasets demonstrate that our method achieves high-fidelity surface reconstruction as well as photorealistic novel view synthesis.

Related papers

• SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes [7.590932716513324]
  We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes.
  arXiv  Detail & Related papers  (2024-10-22T17:59:56Z)
• Learning Topology Uniformed Face Mesh by Volume Rendering for Multi-view Reconstruction [40.45683488053611]
  Face meshes in consistent topology serve as the foundation for many face-related applications. We propose a mesh volume rendering method that enables directly optimizing mesh geometry while preserving topology. Key innovation lies in spreading sparse mesh features into the surrounding space to simulate radiance field required for volume rendering.
  arXiv  Detail & Related papers  (2024-04-08T15:25:50Z)
• ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting [9.145875902703345]
  We introduce ENVIDR, a rendering and modeling framework for high-quality rendering and reconstruction of surfaces with challenging specular reflections. We first propose a novel neural with decomposed rendering to learn the interaction between surface and environment lighting. We then propose an SDF-based neural surface model that leverages this learned neural to represent general scenes.
  arXiv  Detail & Related papers  (2023-03-23T04:12:07Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos [82.74918564737591]
  We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches.
  arXiv  Detail & Related papers  (2022-10-22T04:57:55Z)
• Learning Signed Distance Field for Multi-view Surface Reconstruction [24.090786783370195]
  We introduce a novel neural surface reconstruction framework that leverages the knowledge of stereo matching and feature consistency. We apply a signed distance field (SDF) and a surface light field to represent the scene geometry and appearance respectively. Our method is able to improve the robustness of geometry estimation and support reconstruction of complex scene topologies.
  arXiv  Detail & Related papers  (2021-08-23T06:23:50Z)
• NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction [88.02850205432763]
  We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision. We observe that the conventional volume rendering method causes inherent geometric errors for surface reconstruction. We propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision.
  arXiv  Detail & Related papers  (2021-06-20T12:59:42Z)
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control [80.79820002330457]
  We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
  arXiv  Detail & Related papers  (2021-06-03T17:40:48Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)
• Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction [9.747648609960185]
  We present dynamic neural radiance fields for modeling the appearance and dynamics of a human face. Especially, for telepresence applications in AR or VR, a faithful reproduction of the appearance including novel viewpoints or head-poses is required.
  arXiv  Detail & Related papers  (2020-12-05T16:01:16Z)
• Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes [70.76742458931935]
  We introduce a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion. Our representation is optimized through a neural network to fit the observed input views. We show that our representation can be used for complex dynamic scenes, including thin structures, view-dependent effects, and natural degrees of motion.
  arXiv  Detail & Related papers  (2020-11-26T01:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.