Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering

• URL: http://arxiv.org/abs/2403.14554v1
• Date: Thu, 21 Mar 2024 16:53:03 GMT
• Title: Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering
• Authors: Antoine Guédon, Vincent Lepetit,
• Abstract summary: We propose a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches.
• Score: 22.620566178078008
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose Gaussian Frosting, a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time. Our approach builds on the recent 3D Gaussian Splatting framework, which optimizes a set of 3D Gaussians to approximate a radiance field from images. We propose first extracting a base mesh from Gaussians during optimization, then building and refining an adaptive layer of Gaussians with a variable thickness around the mesh to better capture the fine details and volumetric effects near the surface, such as hair or grass. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. The fuzzier the material, the thicker the frosting. We also introduce a parameterization of the Gaussians to enforce them to stay inside the frosting layer and automatically adjust their parameters when deforming, rescaling, editing or animating the mesh. Our representation allows for efficient rendering using Gaussian splatting, as well as editing and animation by modifying the base mesh. We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches. We will release our code and a web-based viewer as additional contributions. Our project page is the following: https://anttwo.github.io/frosting/

Related papers

• GaussianAvatar-Editor: Photorealistic Animatable Gaussian Head Avatar Editor [53.58201080040624]
  We introduce an innovative framework for text-driven editing of animatable Gaussian head avatars. GaussianAvatar-Editor can be fully controlled in expression, pose, and viewpoint.
  arXiv  Detail & Related papers  (2025-01-17T06:40:20Z)
• Textured Gaussians for Enhanced 3D Scene Appearance Modeling [58.134905268540436]
  3D Gaussian Splatting (3DGS) has emerged as a state-of-the-art 3D reconstruction and rendering technique. We propose a new generalized Gaussian appearance representation that augments each Gaussian with alpha(A), RGB, or RGBA texture maps. We demonstrate image quality improvements over existing methods while using a similar or lower number of Gaussians.
  arXiv  Detail & Related papers  (2024-11-27T18:59:59Z)
• MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering [61.64903786502728]
  We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes. We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound splats. Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR.
  arXiv  Detail & Related papers  (2024-10-11T16:07:59Z)
• Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos [58.22272760132996]
  We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. We propose Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality.
  arXiv  Detail & Related papers  (2024-06-26T19:37:07Z)
• Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos [27.531394287148384]
  We introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh given a single monocular video. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from a video. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians.
  arXiv  Detail & Related papers  (2024-04-18T17:58:16Z)
• GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting [7.121259735505479]
  We introduce the Gaussian Mesh Splatting (GaMeS) model, which allows modification of Gaussian components in a similar way as meshes. We also define Gaussian splats solely based on their location on the mesh, allowing for automatic adjustments in position, scale, and rotation during animation.
  arXiv  Detail & Related papers  (2024-02-02T14:50:23Z)
• Gaussian Grouping: Segment and Edit Anything in 3D Scenes [65.49196142146292]
  We propose Gaussian Grouping, which extends Gaussian Splatting to jointly reconstruct and segment anything in open-world 3D scenes. Compared to the implicit NeRF representation, we show that the grouped 3D Gaussians can reconstruct, segment and edit anything in 3D with high visual quality, fine granularity and efficiency.
  arXiv  Detail & Related papers  (2023-12-01T17:09:31Z)
• GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting [66.08674785436612]
  3D editing plays a crucial role in many areas such as gaming and virtual reality. Traditional 3D editing methods, which rely on representations like meshes and point clouds, often fall short in realistically depicting complex scenes. Our paper presents GaussianEditor, an innovative and efficient 3D editing algorithm based on Gaussian Splatting (GS), a novel 3D representation.
  arXiv  Detail & Related papers  (2023-11-24T14:46:59Z)
• SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering [24.91019554830571]
  We propose a method to allow precise and extremely fast mesh extraction from 3D Gaussian Splatting. It is however challenging to extract a mesh from the millions of tiny 3D gaussians as these gaussians tend to be unorganized after optimization.
  arXiv  Detail & Related papers  (2023-11-21T18:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.