XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies

• URL: http://arxiv.org/abs/2312.03806v2
• Date: Tue, 25 Jun 2024 17:01:54 GMT
• Title: XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies
• Authors: Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, Francis Williams,
• Abstract summary: We present XCube, a novel generative model for high-resolution sparse 3D voxel grids with arbitrary attributes. In addition to generating high-resolution objects, we show that our model can be used to solve a variety of tasks such as user-guided editing, scene completion from a single scan, and text-to-3D.
• Score: 56.460739605550565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present XCube (abbreviated as $\mathcal{X}^3$), a novel generative model for high-resolution sparse 3D voxel grids with arbitrary attributes. Our model can generate millions of voxels with a finest effective resolution of up to $1024^3$ in a feed-forward fashion without time-consuming test-time optimization. To achieve this, we employ a hierarchical voxel latent diffusion model which generates progressively higher resolution grids in a coarse-to-fine manner using a custom framework built on the highly efficient VDB data structure. Apart from generating high-resolution objects, we demonstrate the effectiveness of XCube on large outdoor scenes at scales of 100m$\times$100m with a voxel size as small as 10cm. We observe clear qualitative and quantitative improvements over past approaches. In addition to unconditional generation, we show that our model can be used to solve a variety of tasks such as user-guided editing, scene completion from a single scan, and text-to-3D. The source code and more results can be found at https://research.nvidia.com/labs/toronto-ai/xcube/.

Related papers

• Structured 3D Latents for Scalable and Versatile 3D Generation [28.672494137267837]
  We introduce a novel 3D generation method for versatile and high-quality 3D asset creation. The cornerstone is a unified Structured LATent representation which allows decoding to different output formats. This is achieved by integrating a sparsely-populated 3D grid with dense multiview visual features extracted from a powerful vision foundation model.
  arXiv  Detail & Related papers  (2024-12-02T13:58:38Z)
• SCube: Instant Large-Scale Scene Reconstruction using VoxSplats [55.383993296042526]
  We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold.
  arXiv  Detail & Related papers  (2024-10-26T00:52:46Z)
• LaGeM: A Large Geometry Model for 3D Representation Learning and Diffusion [46.76882780184126]
  This paper introduces a novel hierarchical autoencoder that maps 3D models into a compressed latent space. We show that the model can be used to represent a wide range of 3D models while faithfully representing high-resolution geometry details.
  arXiv  Detail & Related papers  (2024-10-02T07:42:20Z)
• LT3SD: Latent Trees for 3D Scene Diffusion [71.91446143124648]
  We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. We demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation.
  arXiv  Detail & Related papers  (2024-09-12T16:55:51Z)
• Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata [70.9375320609781]
  We aim to generate fine-grained 3D geometry from large-scale sparse LiDAR scans, abundantly captured by autonomous vehicles (AV) We propose hierarchical Generative Cellular Automata (hGCA), a spatially scalable 3D generative model, which grows geometry with local kernels following, in a coarse-to-fine manner, equipped with a light-weight planner to induce global consistency.
  arXiv  Detail & Related papers  (2024-06-12T14:56:56Z)
• SuperGaussian: Repurposing Video Models for 3D Super Resolution [67.19266415499139]
  We present a simple, modular, and generic method that upsamples coarse 3D models by adding geometric and appearance details. We demonstrate that it is possible to directly repurpose existing (pretrained) video models for 3D super-resolution.
  arXiv  Detail & Related papers  (2024-06-02T03:44:50Z)
• Make-A-Shape: a Ten-Million-scale 3D Shape Model [52.701745578415796]
  This paper introduces Make-A-Shape, a new 3D generative model designed for efficient training on a vast scale. We first innovate a wavelet-tree representation to compactly encode shapes by formulating the subband coefficient filtering scheme. We derive the subband adaptive training strategy to train our model to effectively learn to generate coarse and detail wavelet coefficients.
  arXiv  Detail & Related papers  (2024-01-20T00:21:58Z)
• VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids [42.74658047803192]
  State-of-the-art 3D-aware generative models rely on coordinate-based parameterize 3D radiance fields. Existing approaches often render low-resolution feature maps and process them with an upsampling network to obtain the final image. In contrast to existing approaches, our method requires only a single forward pass to generate a full 3D scene.
  arXiv  Detail & Related papers  (2022-06-15T17:44:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.