Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering

• URL: http://arxiv.org/abs/2403.14554v1
• Date: Thu, 21 Mar 2024 16:53:03 GMT
• Title: Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering
• Authors: Antoine Guédon, Vincent Lepetit,
• Abstract summary: We propose a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches.
• Score: 22.620566178078008
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose Gaussian Frosting, a novel mesh-based representation for high-quality rendering and editing of complex 3D effects in real-time. Our approach builds on the recent 3D Gaussian Splatting framework, which optimizes a set of 3D Gaussians to approximate a radiance field from images. We propose first extracting a base mesh from Gaussians during optimization, then building and refining an adaptive layer of Gaussians with a variable thickness around the mesh to better capture the fine details and volumetric effects near the surface, such as hair or grass. We call this layer Gaussian Frosting, as it resembles a coating of frosting on a cake. The fuzzier the material, the thicker the frosting. We also introduce a parameterization of the Gaussians to enforce them to stay inside the frosting layer and automatically adjust their parameters when deforming, rescaling, editing or animating the mesh. Our representation allows for efficient rendering using Gaussian splatting, as well as editing and animation by modifying the base mesh. We demonstrate the effectiveness of our method on various synthetic and real scenes, and show that it outperforms existing surface-based approaches. We will release our code and a web-based viewer as additional contributions. Our project page is the following: https://anttwo.github.io/frosting/

Related papers

• 3D Gaussian Inverse Rendering with Approximated Global Illumination [15.899514468603627]
  We present a novel approach that enables efficient global illumination for 3D Gaussians Splatting through screen-space ray tracing. Our key insight is that a substantial amount of indirect light can be traced back to surfaces visible within the current view frustum. In experiments, we show that the screen-space approximation we utilize allows for indirect illumination and supports real-time rendering and editing.
  arXiv  Detail & Related papers  (2025-04-02T05:02:25Z)
• GaussianAvatar-Editor: Photorealistic Animatable Gaussian Head Avatar Editor [53.58201080040624]
  We introduce an innovative framework for text-driven editing of animatable Gaussian head avatars. GaussianAvatar-Editor can be fully controlled in expression, pose, and viewpoint.
  arXiv  Detail & Related papers  (2025-01-17T06:40:20Z)
• MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering [61.64903786502728]
  We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes. We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound splats. Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR.
  arXiv  Detail & Related papers  (2024-10-11T16:07:59Z)
• Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos [58.22272760132996]
  We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. We propose Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality.
  arXiv  Detail & Related papers  (2024-06-26T19:37:07Z)
• Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos [27.531394287148384]
  We introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh given a single monocular video. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from a video. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians.
  arXiv  Detail & Related papers  (2024-04-18T17:58:16Z)
• DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading [50.331929164207324]
  We introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view and editing tasks.
  arXiv  Detail & Related papers  (2024-04-15T01:58:54Z)
• Bridging 3D Gaussian and Mesh for Freeview Video Rendering [57.21847030980905]
  GauMesh bridges the 3D Gaussian and Mesh for modeling and rendering the dynamic scenes. We show that our approach adapts the appropriate type of primitives to represent the different parts of the dynamic scene.
  arXiv  Detail & Related papers  (2024-03-18T04:01:26Z)
• GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting [11.791944275269266]
  We introduce the Gaussian Mesh Splatting (GaMeS) model, which allows modification of Gaussian components in a similar way as meshes. We also define Gaussian splats solely based on their location on the mesh, allowing for automatic adjustments in position, scale, and rotation during animation.
  arXiv  Detail & Related papers  (2024-02-02T14:50:23Z)
• SplatMesh: Interactive 3D Segmentation and Editing Using Mesh-Based Gaussian Splatting [86.50200613220674]
  A key challenge in 3D-based interactive editing is the absence of an efficient representation that balances diverse modifications with high-quality view synthesis under a given memory constraint. We introduce SplatMesh, a novel fine-grained interactive 3D segmentation and editing algorithm that integrates 3D Gaussian Splatting with a precomputed mesh. By segmenting and editing the simplified mesh, we can effectively edit the Gaussian splats as well, which will lead to extensive experiments on real and synthetic datasets.
  arXiv  Detail & Related papers  (2023-12-26T02:50:42Z)
• Gaussian Grouping: Segment and Edit Anything in 3D Scenes [65.49196142146292]
  We propose Gaussian Grouping, which extends Gaussian Splatting to jointly reconstruct and segment anything in open-world 3D scenes. Compared to the implicit NeRF representation, we show that the grouped 3D Gaussians can reconstruct, segment and edit anything in 3D with high visual quality, fine granularity and efficiency.
  arXiv  Detail & Related papers  (2023-12-01T17:09:31Z)
• GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting [66.08674785436612]
  3D editing plays a crucial role in many areas such as gaming and virtual reality. Traditional 3D editing methods, which rely on representations like meshes and point clouds, often fall short in realistically depicting complex scenes. Our paper presents GaussianEditor, an innovative and efficient 3D editing algorithm based on Gaussian Splatting (GS), a novel 3D representation.
  arXiv  Detail & Related papers  (2023-11-24T14:46:59Z)
• SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering [24.91019554830571]
  We propose a method to allow precise and extremely fast mesh extraction from 3D Gaussian Splatting. It is however challenging to extract a mesh from the millions of tiny 3D gaussians as these gaussians tend to be unorganized after optimization.
  arXiv  Detail & Related papers  (2023-11-21T18:38:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.