MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections
- URL: http://arxiv.org/abs/2405.11921v1
- Date: Mon, 20 May 2024 09:58:03 GMT
- Title: MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections
- Authors: Jiayue Liu, Xiao Tang, Freeman Cheng, Roy Yang, Zhihao Li, Jianzhuang Liu, Yi Huang, Jiaqi Lin, Shiyong Liu, Xiaofei Wu, Songcen Xu, Chun Yuan,
- Abstract summary: MirrorGaussian is the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting.
We introduce an intuitive dual-rendering strategy that enables differentiableization of both the real-world 3D Gaussians and the mirrored counterpart.
Our approach significantly outperforms existing methods, achieving state-of-the-art results.
- Score: 58.003014868772254
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: 3D Gaussian Splatting showcases notable advancements in photo-realistic and real-time novel view synthesis. However, it faces challenges in modeling mirror reflections, which exhibit substantial appearance variations from different viewpoints. To tackle this problem, we present MirrorGaussian, the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting. The key insight is grounded on the mirror symmetry between the real-world space and the virtual mirror space. We introduce an intuitive dual-rendering strategy that enables differentiable rasterization of both the real-world 3D Gaussians and the mirrored counterpart obtained by reflecting the former about the mirror plane. All 3D Gaussians are jointly optimized with the mirror plane in an end-to-end framework. MirrorGaussian achieves high-quality and real-time rendering in scenes with mirrors, empowering scene editing like adding new mirrors and objects. Comprehensive experiments on multiple datasets demonstrate that our approach significantly outperforms existing methods, achieving state-of-the-art results. Project page: https://mirror-gaussian.github.io/.
Related papers
- Mirror-3DGS: Incorporating Mirror Reflections into 3D Gaussian Splatting [27.361324194709155]
Mirror-3DGS is an innovative rendering framework devised to master the intricacies of mirror geometries and reflections.
By incorporating mirror attributes into the 3DGS, Mirror-3DGS crafts a mirrored viewpoint to observe from behind the mirror, enriching the realism of scene renderings.
arXiv Detail & Related papers (2024-04-01T15:16:33Z) - 2D Gaussian Splatting for Geometrically Accurate Radiance Fields [50.056790168812114]
3D Gaussian Splatting (3DGS) has recently revolutionized radiance field reconstruction, achieving high quality novel view synthesis and fast rendering speed without baking.
We present 2D Gaussian Splatting (2DGS), a novel approach to model and reconstruct geometrically accurate radiance fields from multi-view images.
We demonstrate that our differentiable terms allows for noise-free and detailed geometry reconstruction while maintaining competitive appearance quality, fast training speed, and real-time rendering.
arXiv Detail & Related papers (2024-03-26T17:21:24Z) - FDGaussian: Fast Gaussian Splatting from Single Image via Geometric-aware Diffusion Model [81.03553265684184]
We introduce FDGaussian, a novel two-stage framework for single-image 3D reconstruction.
Recent methods typically utilize pre-trained 2D diffusion models to generate plausible novel views from the input image.
We demonstrate that FDGaussian generates images with high consistency across different views and reconstructs high-quality 3D objects.
arXiv Detail & Related papers (2024-03-15T12:24:36Z) - GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorization [76.52007427483396]
GIR is a 3D Gaussian Inverse Rendering method for relightable scene factorization.
Our method utilizes 3D Gaussians to estimate the material properties, illumination, and geometry of an object from multi-view images.
arXiv Detail & Related papers (2023-12-08T16:05:15Z) - GaussianShader: 3D Gaussian Splatting with Shading Functions for
Reflective Surfaces [45.15827491185572]
We present a novel method that applies a simplified shading function on 3D Gaussians to enhance the neural rendering in scenes with reflective surfaces.
Experiments show that GaussianShader strikes a commendable balance between efficiency and visual quality.
arXiv Detail & Related papers (2023-11-29T17:22:26Z) - Mirror-Aware Neural Humans [21.0548144424571]
We develop a consumer-level 3D motion capture system that starts from off-the-shelf 2D poses by automatically calibrating the camera.
We empirically demonstrate the benefit of learning a body model and accounting for occlusion in challenging mirror scenes.
arXiv Detail & Related papers (2023-09-09T10:43:45Z) - Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with
Whitted-Style Ray Tracing [33.852910220413655]
We present a novel neural rendering framework, named Mirror-NeRF, which is able to learn accurate geometry and reflection of the mirror.
Mirror-NeRF supports various scene manipulation applications with mirrors, such as adding new objects or mirrors into the scene and synthesizing the reflections of these new objects in mirrors.
arXiv Detail & Related papers (2023-08-07T03:48:07Z) - Symmetry-Aware Transformer-based Mirror Detection [85.47570468668955]
We propose a dual-path Symmetry-Aware Transformer-based mirror detection Network (SATNet)
SATNet includes two novel modules: Symmetry-Aware Attention Module (SAAM) and Contrast and Fusion Decoder Module (CFDM)
Experimental results show that SATNet outperforms both RGB and RGB-D mirror detection methods on all available mirror detection datasets.
arXiv Detail & Related papers (2022-07-13T16:40:01Z) - Inverting Generative Adversarial Renderer for Face Reconstruction [58.45125455811038]
In this work, we introduce a novel Generative Adversa Renderer (GAR)
GAR learns to model the complicated real-world image, instead of relying on the graphics rules, it is capable of producing realistic images.
Our method achieves state-of-the-art performances on multiple face reconstruction.
arXiv Detail & Related papers (2021-05-06T04:16:06Z) - Structure of Multiple Mirror System from Kaleidoscopic Projections of
Single 3D Point [14.345346642066511]
This paper proposes a novel algorithm of discovering the structure of a kaleidoscopic imaging system that consists of multiple planar mirrors and a camera.
The key contribution of this paper is to propose novel algorithms for these problems using a single 3D point of unknown geometry.
arXiv Detail & Related papers (2021-03-29T11:12:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.