Shap-E: Generating Conditional 3D Implicit Functions

• URL: http://arxiv.org/abs/2305.02463v1
• Date: Wed, 3 May 2023 23:59:13 GMT
• Title: Shap-E: Generating Conditional 3D Implicit Functions
• Authors: Heewoo Jun, Alex Nichol
• Abstract summary: Shap-E is a conditional generative model for 3D assets. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds.
• Score: 7.603750555294962
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space. We release model weights, inference code, and samples at https://github.com/openai/shap-e.

Related papers

• SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images [49.7344030427291]
  We study the problem of single-image 3D object reconstruction. Recent works have diverged into two directions: regression-based modeling and generative modeling. We present SPAR3D, a novel two-stage approach aiming to take the best of both directions.
  arXiv  Detail & Related papers  (2025-01-08T18:52:03Z)
• Structured 3D Latents for Scalable and Versatile 3D Generation [28.672494137267837]
  We introduce a novel 3D generation method for versatile and high-quality 3D asset creation. The cornerstone is a unified Structured LATent representation which allows decoding to different output formats. This is achieved by integrating a sparsely-populated 3D grid with dense multiview visual features extracted from a powerful vision foundation model.
  arXiv  Detail & Related papers  (2024-12-02T13:58:38Z)
• GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction [52.04103235260539]
  We present a diffusion model approach based on Gaussian Splatting representation for 3D object reconstruction from a single view. The model learns to generate 3D objects represented by sets of GS ellipsoids. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views.
  arXiv  Detail & Related papers  (2024-07-05T03:43:08Z)
• DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data [50.164670363633704]
  We present DIRECT-3D, a diffusion-based 3D generative model for creating high-quality 3D assets from text prompts. Our model is directly trained on extensive noisy and unaligned in-the-wild' 3D assets. We achieve state-of-the-art performance in both single-class generation and text-to-3D generation.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• SuperGaussian: Repurposing Video Models for 3D Super Resolution [67.19266415499139]
  We present a simple, modular, and generic method that upsamples coarse 3D models by adding geometric and appearance details. We demonstrate that it is possible to directly repurpose existing (pretrained) video models for 3D super-resolution.
  arXiv  Detail & Related papers  (2024-06-02T03:44:50Z)
• GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models [102.22388340738536]
  2D and 3D diffusion models can generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation.
  arXiv  Detail & Related papers  (2023-10-12T17:22:24Z)
• HoloDiffusion: Training a 3D Diffusion Model using 2D Images [71.1144397510333]
  We introduce a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.
  arXiv  Detail & Related papers  (2023-03-29T07:35:56Z)
• Zero3D: Semantic-Driven Multi-Category 3D Shape Generation [15.665757749429702]
  Previous works face problems with single-category generation, low-frequency 3D details, and requiring a large number of paired datasets for training. To tackle these challenges, we propose a multi-category conditional diffusion model. We employ the hidden-layer diffusion model conditioned on the multi-category shape vector, which greatly reduces the training time and memory consumption.
  arXiv  Detail & Related papers  (2023-01-31T12:43:54Z)
• 3D Neural Field Generation using Triplane Diffusion [37.46688195622667]
  We present an efficient diffusion-based model for 3D-aware generation of neural fields. Our approach pre-processes training data, such as ShapeNet meshes, by converting them to continuous occupancy fields. We demonstrate state-of-the-art results on 3D generation on several object classes from ShapeNet.
  arXiv  Detail & Related papers  (2022-11-30T01:55:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.