Textured Gaussians for Enhanced 3D Scene Appearance Modeling

• URL: http://arxiv.org/abs/2411.18625v1
• Date: Wed, 27 Nov 2024 18:59:59 GMT
• Title: Textured Gaussians for Enhanced 3D Scene Appearance Modeling
• Authors: Brian Chao, Hung-Yu Tseng, Lorenzo Porzi, Chen Gao, Tuotuo Li, Qinbo Li, Ayush Saraf, Jia-Bin Huang, Johannes Kopf, Gordon Wetzstein, Changil Kim,
• Abstract summary: 3D Gaussian Splatting (3DGS) has emerged as a state-of-the-art 3D reconstruction and rendering technique. We propose a new generalized Gaussian appearance representation that augments each Gaussian with alpha(A), RGB, or RGBA texture maps. We demonstrate image quality improvements over existing methods while using a similar or lower number of Gaussians.
• Score: 58.134905268540436
• License:
• Abstract: 3D Gaussian Splatting (3DGS) has recently emerged as a state-of-the-art 3D reconstruction and rendering technique due to its high-quality results and fast training and rendering time. However, pixels covered by the same Gaussian are always shaded in the same color up to a Gaussian falloff scaling factor. Furthermore, the finest geometric detail any individual Gaussian can represent is a simple ellipsoid. These properties of 3DGS greatly limit the expressivity of individual Gaussian primitives. To address these issues, we draw inspiration from texture and alpha mapping in traditional graphics and integrate it with 3DGS. Specifically, we propose a new generalized Gaussian appearance representation that augments each Gaussian with alpha~(A), RGB, or RGBA texture maps to model spatially varying color and opacity across the extent of each Gaussian. As such, each Gaussian can represent a richer set of texture patterns and geometric structures, instead of just a single color and ellipsoid as in naive Gaussian Splatting. Surprisingly, we found that the expressivity of Gaussians can be greatly improved by using alpha-only texture maps, and further augmenting Gaussians with RGB texture maps achieves the highest expressivity. We validate our method on a wide variety of standard benchmark datasets and our own custom captures at both the object and scene levels. We demonstrate image quality improvements over existing methods while using a similar or lower number of Gaussians.

Related papers

• Pushing Rendering Boundaries: Hard Gaussian Splatting [72.28941128988292]
  3D Gaussian Splatting (3DGS) has demonstrated impressive Novel View Synthesis (NVS) results in a real-time rendering manner. We propose Hard Gaussian Splatting, dubbed HGS, which considers multi-view significant positional gradients and rendering errors to grow hard Gaussians. Our method achieves state-of-the-art rendering quality while maintaining real-time efficiency.
  arXiv  Detail & Related papers  (2024-12-06T07:42:47Z)
• GStex: Per-Primitive Texturing of 2D Gaussian Splatting for Decoupled Appearance and Geometry Modeling [None]
  Gaussian splatting has demonstrated excellent performance for view synthesis and scene reconstruction. Since each Gaussian primitive encodes both appearance and geometry, appearance modeling requires a number of Gaussian primitives. We propose to employ perprimitive representation so that even a single Gaussian can be used to capture appearance details.
  arXiv  Detail & Related papers  (2024-09-19T17:58:44Z)
• Textured-GS: Gaussian Splatting with Spatially Defined Color and Opacity [7.861993966048637]
  We introduce Textured-GS, an innovative method for rendering Gaussian splatting using Spherical Harmonics (SH) This approach enables each Gaussian to exhibit a richer representation by accommodating varying colors and opacities across its surface. Our experiments show that Textured-GS consistently outperforms both the baseline Mini-Splatting and standard 3DGS in terms of visual fidelity.
  arXiv  Detail & Related papers  (2024-07-13T00:45:37Z)
• GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality [99.63429416013713]
  3D-GS has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. We propose a novel framework named GaussianDreamerPro to enhance the generation quality.
  arXiv  Detail & Related papers  (2024-06-26T16:12:09Z)
• PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting [59.277480452459315]
  We propose a principled sensitivity pruning score that preserves visual fidelity and foreground details at significantly higher compression ratios. We also propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model without changing its training pipeline.
  arXiv  Detail & Related papers  (2024-06-14T17:53:55Z)
• GSGAN: Adversarial Learning for Hierarchical Generation of 3D Gaussian Splats [20.833116566243408]
  In this paper, we exploit Gaussian as a 3D representation for 3D GANs by leveraging its efficient and explicit characteristics. We introduce a generator architecture with a hierarchical multi-scale Gaussian representation that effectively regularizes the position and scale of generated Gaussians. Experimental results demonstrate that ours achieves a significantly faster rendering speed (x100) compared to state-of-the-art 3D consistent GANs.
  arXiv  Detail & Related papers  (2024-06-05T05:52:20Z)
• RTG-SLAM: Real-time 3D Reconstruction at Scale using Gaussian Splatting [51.51310922527121]
  We present a real-time 3D reconstruction system with an RGBD camera for large-scale environments using Gaussian splatting. We force each Gaussian to be either opaque or nearly transparent, with the opaque ones fitting the surface and dominant colors, and transparent ones fitting residual colors. We show real-time reconstructions of a variety of large scenes and show superior performance in the realism of novel view synthesis and camera tracking accuracy.
  arXiv  Detail & Related papers  (2024-04-30T16:54:59Z)
• LeanGaussian: Breaking Pixel or Point Cloud Correspondence in Modeling 3D Gaussians [12.408610403423559]
  We introduce LeanGaussian, a novel approach that treats each query in deformable Transformer as one 3D Gaussian ellipsoid. We leverage deformable decoder to iteratively refine the Gaussians layer-by-layer with the image features as keys and values. Our approach outperforms prior methods by approximately 6.1%, achieving a PSNR of 25.44 and 22.36, respectively.
  arXiv  Detail & Related papers  (2024-04-25T04:18:59Z)
• GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning [60.33970027554299]
  Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable avatars from textual descriptions. Our proposed method, GAvatar, enables the large-scale generation of diverse animatable avatars using only text prompts.
  arXiv  Detail & Related papers  (2023-12-18T18:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.