GaussianCity: Generative Gaussian Splatting for Unbounded 3D City Generation

• URL: http://arxiv.org/abs/2406.06526v2
• Date: Thu, 14 Nov 2024 07:29:33 GMT
• Title: GaussianCity: Generative Gaussian Splatting for Unbounded 3D City Generation
• Authors: Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu,
• Abstract summary: 3D Gaussian Splatting (3D-GS) has emerged as a highly efficient alternative for object-level 3D generation. However, adapting 3D-GS from finite-scale 3D objects and humans to infinite-scale 3D cities is non-trivial. We propose a generative Gaussian Splatting framework dedicated to efficiently synthesizing 3D cities with a single feed-forward pass.
• Score: 44.203932215464214
• License:
• Abstract: 3D city generation with NeRF-based methods shows promising generation results but is computationally inefficient. Recently 3D Gaussian Splatting (3D-GS) has emerged as a highly efficient alternative for object-level 3D generation. However, adapting 3D-GS from finite-scale 3D objects and humans to infinite-scale 3D cities is non-trivial. Unbounded 3D city generation entails significant storage overhead (out-of-memory issues), arising from the need to expand points to billions, often demanding hundreds of Gigabytes of VRAM for a city scene spanning 10km^2. In this paper, we propose GaussianCity, a generative Gaussian Splatting framework dedicated to efficiently synthesizing unbounded 3D cities with a single feed-forward pass. Our key insights are two-fold: 1) Compact 3D Scene Representation: We introduce BEV-Point as a highly compact intermediate representation, ensuring that the growth in VRAM usage for unbounded scenes remains constant, thus enabling unbounded city generation. 2) Spatial-aware Gaussian Attribute Decoder: We present spatial-aware BEV-Point decoder to produce 3D Gaussian attributes, which leverages Point Serializer to integrate the structural and contextual characteristics of BEV points. Extensive experiments demonstrate that GaussianCity achieves state-of-the-art results in both drone-view and street-view 3D city generation. Notably, compared to CityDreamer, GaussianCity exhibits superior performance with a speedup of 60 times (10.72 FPS v.s. 0.18 FPS).

Related papers

• 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes [87.01284850604495]
  We introduce 3D Convexting (3DCS), which leverages 3D smooth convexes as primitives for modeling geometrically-meaningful radiance fields from multiview images. 3DCS achieves superior performance over 3DGS on benchmarks such as MipNeizer, Tanks and Temples, and Deep Blending. Our results highlight the potential of 3D Convexting to become the new standard for high-quality scene reconstruction.
  arXiv  Detail & Related papers  (2024-11-22T14:31:39Z)
• PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting [59.277480452459315]
  We propose a principled spatial sensitivity pruning score that outperforms current approaches. We also propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model. Our pipeline increases the average rendering speed of 3D-GS by 2.65$times$ while retaining more salient foreground information.
  arXiv  Detail & Related papers  (2024-06-14T17:53:55Z)
• GSGAN: Adversarial Learning for Hierarchical Generation of 3D Gaussian Splats [20.833116566243408]
  In this paper, we exploit Gaussian as a 3D representation for 3D GANs by leveraging its efficient and explicit characteristics. We introduce a generator architecture with a hierarchical multi-scale Gaussian representation that effectively regularizes the position and scale of generated Gaussians. Experimental results demonstrate that ours achieves a significantly faster rendering speed (x100) compared to state-of-the-art 3D consistent GANs.
  arXiv  Detail & Related papers  (2024-06-05T05:52:20Z)
• GVGEN: Text-to-3D Generation with Volumetric Representation [89.55687129165256]
  3D Gaussian splatting has emerged as a powerful technique for 3D reconstruction and generation, known for its fast and high-quality rendering capabilities. This paper introduces a novel diffusion-based framework, GVGEN, designed to efficiently generate 3D Gaussian representations from text input.
  arXiv  Detail & Related papers  (2024-03-19T17:57:52Z)
• AGG: Amortized Generative 3D Gaussians for Single Image to 3D [108.38567665695027]
  We introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image. AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. We propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module.
  arXiv  Detail & Related papers  (2024-01-08T18:56:33Z)
• Compact 3D Scene Representation via Self-Organizing Gaussian Grids [10.816451552362823]
  3D Gaussian Splatting has recently emerged as a highly promising technique for modeling of static 3D scenes. We introduce a compact scene representation organizing the parameters of 3DGS into a 2D grid with local homogeneity. Our method achieves a reduction factor of 17x to 42x in size for complex scenes with no increase in training time.
  arXiv  Detail & Related papers  (2023-12-19T20:18:29Z)
• HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting [113.37908093915837]
  Existing methods optimize 3D representations like mesh or neural fields via score distillation sampling (SDS), which suffers from inadequate fine details or excessive training time. In this paper, we propose an efficient yet effective framework, HumanGaussian, that generates high-quality 3D humans with fine-grained geometry and realistic appearance.
  arXiv  Detail & Related papers  (2023-11-28T18:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.