SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes

• URL: http://arxiv.org/abs/2410.17249v2
• Date: Sat, 07 Dec 2024 18:56:01 GMT
• Title: SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes
• Authors: Cheng-De Fan, Chen-Wei Chang, Yi-Ruei Liu, Jie-Ying Lee, Jiun-Long Huang, Yu-Chee Tseng, Yu-Lun Liu,
• Abstract summary: We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. It is the only existing 3DGS method capable of synthesizing real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes.
• Score: 7.590932716513324
• License:
• Abstract: We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. Previous methods extending 3DGS to model dynamic scenes have struggled to represent specular surfaces accurately. Our method addresses this limitation by introducing a residual correction technique for accurate surface normal computation during deformation, complemented by a deformable environment map that adapts to time-varying lighting conditions. We implement a coarse-to-fine training strategy that significantly enhances scene geometry and specular color prediction. It is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes. See our project page for video results at https://cdfan0627.github.io/spectromotion/.

Related papers

• MVS-GS: High-Quality 3D Gaussian Splatting Mapping via Online Multi-View Stereo [9.740087094317735]
  We propose a novel framework for high-quality 3DGS modeling using an online multi-view stereo approach. Our method estimates MVS depth using sequential frames from a local time window and applies comprehensive depth refinement techniques. Experimental results demonstrate that our method outperforms state-of-the-art dense SLAM methods.
  arXiv  Detail & Related papers  (2024-12-26T09:20:04Z)
• Modeling Ambient Scene Dynamics for Free-view Synthesis [31.233859111566613]
  We introduce a novel method for dynamic free-view synthesis of an ambient scenes from a monocular capture. Our method builds upon the recent advancements in 3D Gaussian Splatting (3DGS) that can faithfully reconstruct complex static scenes.
  arXiv  Detail & Related papers  (2024-06-13T17:59:11Z)
• Dynamic 3D Gaussian Fields for Urban Areas [60.64840836584623]
  We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. We propose 4DGF, a neural scene representation that scales to large-scale dynamic urban areas.
  arXiv  Detail & Related papers  (2024-06-05T12:07:39Z)
• SAGS: Structure-Aware 3D Gaussian Splatting [53.6730827668389]
  We propose a structure-aware Gaussian Splatting method (SAGS) that implicitly encodes the geometry of the scene. SAGS reflects to state-of-the-art rendering performance and reduced storage requirements on benchmark novel-view synthesis datasets.
  arXiv  Detail & Related papers  (2024-04-29T23:26:30Z)
• SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting [7.553079256251747]
  We extend 3D Gaussian Splatting to reconstruct dynamic scenes. We produce high-quality renderings of general dynamic scenes with competitive quantitative performance. Our method can be viewed in real-time in our dynamic interactive viewer.
  arXiv  Detail & Related papers  (2023-12-20T03:54:03Z)
• GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesis [16.733855781461802]
  Implicit deformable representations commonly model motion with a canonical space and time-dependent deformation field. GauFRe, uses a forward-warping deformation to explicitly model non-rigid transformations of scene geometry. Experiments show our method achieves competitive results and higher efficiency than previous state-of-the-art NeRF and Gaussian-based methods.
  arXiv  Detail & Related papers  (2023-12-18T18:59:03Z)
• SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes [59.23385953161328]
  Novel view synthesis for dynamic scenes is still a challenging problem in computer vision and graphics. We propose a new representation that explicitly decomposes the motion and appearance of dynamic scenes into sparse control points and dense Gaussians. Our method can enable user-controlled motion editing while retaining high-fidelity appearances.
  arXiv  Detail & Related papers  (2023-12-04T11:57:14Z)
• Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction [29.83056271799794]
  Implicit neural representation has paved the way for new approaches to dynamic scene reconstruction and rendering. We propose a deformable 3D Gaussians Splatting method that reconstructs scenes using 3D Gaussians and learns them in canonical space. Through a differential Gaussianizer, the deformable 3D Gaussians not only achieve higher rendering quality but also real-time rendering speed.
  arXiv  Detail & Related papers  (2023-09-22T16:04:02Z)
• NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos [82.74918564737591]
  We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches.
  arXiv  Detail & Related papers  (2022-10-22T04:57:55Z)
• Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model [76.64071133839862]
  Capturing general deforming scenes from monocular RGB video is crucial for many computer graphics and vision applications. Our method, Ub4D, handles large deformations, performs shape completion in occluded regions, and can operate on monocular RGB videos directly by using differentiable volume rendering. Results on our new dataset, which will be made publicly available, demonstrate a clear improvement over the state of the art in terms of surface reconstruction accuracy and robustness to large deformations.
  arXiv  Detail & Related papers  (2022-06-16T17:59:54Z)
• Neural Deformable Voxel Grid for Fast Optimization of Dynamic View Synthesis [63.25919018001152]
  We propose a fast deformable radiance field method to handle dynamic scenes. Our method achieves comparable performance to D-NeRF using only 20 minutes for training.
  arXiv  Detail & Related papers  (2022-06-15T17:49:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.