GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation

• URL: http://arxiv.org/abs/2411.08033v1
• Date: Tue, 12 Nov 2024 18:59:32 GMT
• Title: GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation
• Authors: Yushi Lan, Shangchen Zhou, Zhaoyang Lyu, Fangzhou Hong, Shuai Yang, Bo Dai, Xingang Pan, Chen Change Loy,
• Abstract summary: This paper introduces a novel 3D generation framework, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single/multi-view image inputs.
• Score: 75.39457097832113
• License:
• Abstract: While 3D content generation has advanced significantly, existing methods still face challenges with input formats, latent space design, and output representations. This paper introduces a novel 3D generation framework that addresses these challenges, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder (VAE) with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information, and incorporates a cascaded latent diffusion model for improved shape-texture disentanglement. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single/multi-view image inputs. Notably, the newly proposed latent space naturally enables geometry-texture disentanglement, thus allowing 3D-aware editing. Experimental results demonstrate the effectiveness of our approach on multiple datasets, outperforming existing methods in both text- and image-conditioned 3D generation.

Related papers

• LAM3D: Large Image-Point-Cloud Alignment Model for 3D Reconstruction from Single Image [64.94932577552458]
  Large Reconstruction Models have made significant strides in the realm of automated 3D content generation from single or multiple input images. Despite their success, these models often produce 3D meshes with geometric inaccuracies, stemming from the inherent challenges of deducing 3D shapes solely from image data. We introduce a novel framework, the Large Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud data to enhance the fidelity of generated 3D meshes.
  arXiv  Detail & Related papers  (2024-05-24T15:09:12Z)
• NeuSDFusion: A Spatial-Aware Generative Model for 3D Shape Completion, Reconstruction, and Generation [52.772319840580074]
  3D shape generation aims to produce innovative 3D content adhering to specific conditions and constraints. Existing methods often decompose 3D shapes into a sequence of localized components, treating each element in isolation. We introduce a novel spatial-aware 3D shape generation framework that leverages 2D plane representations for enhanced 3D shape modeling.
  arXiv  Detail & Related papers  (2024-03-27T04:09:34Z)
• Compress3D: a Compressed Latent Space for 3D Generation from a Single Image [27.53099431097921]
  Triplane autoencoder encodes 3D models into a compact triplane latent space to compress both the 3D geometry and texture information. We introduce a 3D-aware cross-attention mechanism, which utilizes low-resolution latent representations to query features from a high-resolution 3D feature volume. Our approach enables the generation of high-quality 3D assets in merely 7 seconds on a single A100 GPU.
  arXiv  Detail & Related papers  (2024-03-20T11:51:04Z)
• LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation [73.36690511083894]
  This paper introduces a novel framework called LN3Diff to address a unified 3D diffusion pipeline. Our approach harnesses a 3D-aware architecture and variational autoencoder to encode the input image into a structured, compact, and 3D latent space. It achieves state-of-the-art performance on ShapeNet for 3D generation and demonstrates superior performance in monocular 3D reconstruction and conditional 3D generation.
  arXiv  Detail & Related papers  (2024-03-18T17:54:34Z)
• Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability [118.26563926533517]
  Auto-regressive models have achieved impressive results in 2D image generation by modeling joint distributions in grid space. We extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously.
  arXiv  Detail & Related papers  (2024-02-19T15:33:09Z)
• Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation [29.818827785812086]
  Controllable generation of 3D assets is important for many practical applications like content creation in movies, games and engineering, as well as in AR/VR. We present a suitable representation for 3D diffusion models to enable disentanglement by introducing a hybrid point cloud and neural radiance field approach.
  arXiv  Detail & Related papers  (2023-12-21T18:46:27Z)
• Pushing the Limits of 3D Shape Generation at Scale [65.24420181727615]
  We present a significant breakthrough in 3D shape generation by scaling it to unprecedented dimensions. We have developed a model with an astounding 3.6 billion trainable parameters, establishing it as the largest 3D shape generation model to date, named Argus-3D.
  arXiv  Detail & Related papers  (2023-06-20T13:01:19Z)
• 3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models [8.583859530633417]
  We propose a diffusion model for neural implicit representations of 3D shapes that operates in the latent space of an auto-decoder. This allows us to generate diverse and high quality 3D surfaces.
  arXiv  Detail & Related papers  (2022-12-01T20:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.