HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian
Splatting
- URL: http://arxiv.org/abs/2312.03461v2
- Date: Thu, 7 Dec 2023 12:46:07 GMT
- Title: HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian
Splatting
- Authors: Yuheng Jiang, Zhehao Shen, Penghao Wang, Zhuo Su, Yu Hong, Yingliang
Zhang, Jingyi Yu, Lan Xu
- Abstract summary: HiFi4G is an explicit and compact Gaussian-based approach for high-fidelity human performance rendering from dense footage.
It achieves a substantial compression rate of approximately 25 times, with less than 2MB of storage per frame.
- Score: 48.59338619051709
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We have recently seen tremendous progress in photo-real human modeling and
rendering. Yet, efficiently rendering realistic human performance and
integrating it into the rasterization pipeline remains challenging. In this
paper, we present HiFi4G, an explicit and compact Gaussian-based approach for
high-fidelity human performance rendering from dense footage. Our core
intuition is to marry the 3D Gaussian representation with non-rigid tracking,
achieving a compact and compression-friendly representation. We first propose a
dual-graph mechanism to obtain motion priors, with a coarse deformation graph
for effective initialization and a fine-grained Gaussian graph to enforce
subsequent constraints. Then, we utilize a 4D Gaussian optimization scheme with
adaptive spatial-temporal regularizers to effectively balance the non-rigid
prior and Gaussian updating. We also present a companion compression scheme
with residual compensation for immersive experiences on various platforms. It
achieves a substantial compression rate of approximately 25 times, with less
than 2MB of storage per frame. Extensive experiments demonstrate the
effectiveness of our approach, which significantly outperforms existing
approaches in terms of optimization speed, rendering quality, and storage
overhead.
Related papers
- LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstruction [33.794584735264884]
We introduce a Lightweight 4D Gaussian Splatting framework (LGS) for dynamic endoscopic reconstruction.
To minimize the redundancy of Gaussian quantities, we propose Deformation-Aware Pruning.
We also simplify the representation of textures and lighting in non-crucial areas by pruning the dimensions of Gaussian attributes.
arXiv Detail & Related papers (2024-06-23T10:49:39Z) - CompGS: Efficient 3D Scene Representation via Compressed Gaussian Splatting [68.94594215660473]
We propose an efficient 3D scene representation, named Compressed Gaussian Splatting (CompGS)
We exploit a small set of anchor primitives for prediction, allowing the majority of primitives to be encapsulated into highly compact residual forms.
Experimental results show that the proposed CompGS significantly outperforms existing methods, achieving superior compactness in 3D scene representation without compromising model accuracy and rendering quality.
arXiv Detail & Related papers (2024-04-15T04:50:39Z) - CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians [18.42203035154126]
We introduce a structured Gaussian representation that can be controlled in 2D image space.
We then constraint the Gaussians, in particular their position, and prevent them from moving independently during optimization.
We demonstrate significant improvements compared to the state-of-the-art sparse-view NeRF-based approaches on a variety of scenes.
arXiv Detail & Related papers (2024-03-28T15:27:13Z) - VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction [59.40711222096875]
We present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting.
Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets.
arXiv Detail & Related papers (2024-02-27T11:40:50Z) - GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering [112.16239342037714]
GES (Generalized Exponential Splatting) is a novel representation that employs Generalized Exponential Function (GEF) to model 3D scenes.
With the aid of a frequency-modulated loss, GES achieves competitive performance in novel-view synthesis benchmarks.
arXiv Detail & Related papers (2024-02-15T17:32:50Z) - StopThePop: Sorted Gaussian Splatting for View-Consistent Real-time Rendering [42.91830228828405]
We present a novel hierarchicalization approach that culls splats with minimal processing overhead.
Our approach is only 4% slower on average than the original Gaussian Splatting.
rendering performance is nearly doubled, making our approach 1.6x faster than the original Gaussian Splatting.
arXiv Detail & Related papers (2024-02-01T11:46:44Z) - EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene
Reconstruction [36.35631592019182]
We introduce EndoGaussian, a real-time endoscopic scene reconstruction framework built on 3D Gaussian Splatting (3DGS)
Our framework significantly boosts the rendering speed to a real-time level.
Experiments on public datasets demonstrate our efficacy against prior SOTAs in many aspects.
arXiv Detail & Related papers (2024-01-23T08:44:26Z) - GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting [51.96353586773191]
We introduce textbfGS-SLAM that first utilizes 3D Gaussian representation in the Simultaneous Localization and Mapping system.
Our method utilizes a real-time differentiable splatting rendering pipeline that offers significant speedup to map optimization and RGB-D rendering.
Our method achieves competitive performance compared with existing state-of-the-art real-time methods on the Replica, TUM-RGBD datasets.
arXiv Detail & Related papers (2023-11-20T12:08:23Z) - DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation [55.661467968178066]
We propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously.
Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space.
In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks.
arXiv Detail & Related papers (2023-09-28T17:55:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.