End-to-End Rate-Distortion Optimized 3D Gaussian Representation

• URL: http://arxiv.org/abs/2406.01597v2
• Date: Mon, 21 Oct 2024 03:11:29 GMT
• Title: End-to-End Rate-Distortion Optimized 3D Gaussian Representation
• Authors: Henan Wang, Hanxin Zhu, Tianyu He, Runsen Feng, Jiajun Deng, Jiang Bian, Zhibo Chen,
• Abstract summary: We formulate the compact 3D Gaussian learning as an end-to-end Rate-Distortion Optimization problem. We introduce dynamic pruning and entropy-constrained vector quantization (ECVQ) that optimize the rate and distortion at the same time. We verify our method on both real and synthetic scenes, showcasing that RDO-Gaussian greatly reduces the size of 3D Gaussian over 40x.
• Score: 33.20840558425759
• License:
• Abstract: 3D Gaussian Splatting (3DGS) has become an emerging technique with remarkable potential in 3D representation and image rendering. However, the substantial storage overhead of 3DGS significantly impedes its practical applications. In this work, we formulate the compact 3D Gaussian learning as an end-to-end Rate-Distortion Optimization (RDO) problem and propose RDO-Gaussian that can achieve flexible and continuous rate control. RDO-Gaussian addresses two main issues that exist in current schemes: 1) Different from prior endeavors that minimize the rate under the fixed distortion, we introduce dynamic pruning and entropy-constrained vector quantization (ECVQ) that optimize the rate and distortion at the same time. 2) Previous works treat the colors of each Gaussian equally, while we model the colors of different regions and materials with learnable numbers of parameters. We verify our method on both real and synthetic scenes, showcasing that RDO-Gaussian greatly reduces the size of 3D Gaussian over 40x, and surpasses existing methods in rate-distortion performance.

Related papers

• ResGS: Residual Densification of 3D Gaussian for Efficient Detail Recovery [11.706262924395768]
  3D-GS often struggles to capture rich details and complete geometry. We introduce a novel densification method, residual split, which adds a downscaled Gaussian as a residual. Our approach is capable of adaptively retrieving details and complementing missing geometry while enabling progressive refinement.
  arXiv  Detail & Related papers  (2024-12-10T13:19:27Z)
• L3DG: Latent 3D Gaussian Diffusion [74.36431175937285]
  L3DG is the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation. We employ a sparse convolutional architecture to efficiently operate on room-scale scenes. By leveraging the 3D Gaussian representation, the generated scenes can be rendered from arbitrary viewpoints in real-time.
  arXiv  Detail & Related papers  (2024-10-17T13:19:32Z)
• PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting [59.277480452459315]
  We propose a principled sensitivity pruning score that preserves visual fidelity and foreground details at significantly higher compression ratios. We also propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model without changing its training pipeline.
  arXiv  Detail & Related papers  (2024-06-14T17:53:55Z)
• R$^2$-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction [53.19869886963333]
  3D Gaussian splatting (3DGS) has shown promising results in rendering image and surface reconstruction. This paper introduces R2$-Gaussian, the first 3DGS-based framework for sparse-view tomographic reconstruction.
  arXiv  Detail & Related papers  (2024-05-31T08:39:02Z)
• Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes [50.92217884840301]
  Gaussian Opacity Fields (GOF) is a novel approach for efficient, high-quality, and adaptive surface reconstruction in scenes. GOF is derived from ray-tracing-based volume rendering of 3D Gaussians. GOF surpasses existing 3DGS-based methods in surface reconstruction and novel view synthesis.
  arXiv  Detail & Related papers  (2024-04-16T17:57:19Z)
• CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians [18.42203035154126]
  We introduce a structured Gaussian representation that can be controlled in 2D image space. We then constraint the Gaussians, in particular their position, and prevent them from moving independently during optimization. We demonstrate significant improvements compared to the state-of-the-art sparse-view NeRF-based approaches on a variety of scenes.
  arXiv  Detail & Related papers  (2024-03-28T15:27:13Z)
• GaussianPro: 3D Gaussian Splatting with Progressive Propagation [49.918797726059545]
  3DGS relies heavily on the point cloud produced by Structure-from-Motion (SfM) techniques. We propose a novel method that applies a progressive propagation strategy to guide the densification of the 3D Gaussians. Our method significantly surpasses 3DGS on the dataset, exhibiting an improvement of 1.15dB in terms of PSNR.
  arXiv  Detail & Related papers  (2024-02-22T16:00:20Z)
• GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering [112.16239342037714]
  GES (Generalized Exponential Splatting) is a novel representation that employs Generalized Exponential Function (GEF) to model 3D scenes. With the aid of a frequency-modulated loss, GES achieves competitive performance in novel-view synthesis benchmarks.
  arXiv  Detail & Related papers  (2024-02-15T17:32:50Z)
• Sparse-view CT Reconstruction with 3D Gaussian Volumetric Representation [13.667470059238607]
  Sparse-view CT is a promising strategy for reducing the radiation dose of traditional CT scans. Recently, 3D Gaussian has been applied to model complex natural scenes. We investigate their potential for sparse-view CT reconstruction.
  arXiv  Detail & Related papers  (2023-12-25T09:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.