TPA3D: Triplane Attention for Fast Text-to-3D Generation

• URL: http://arxiv.org/abs/2312.02647v4
• Date: Mon, 30 Sep 2024 13:41:51 GMT
• Title: TPA3D: Triplane Attention for Fast Text-to-3D Generation
• Authors: Bin-Shih Wu, Hong-En Chen, Sheng-Yu Huang, Yu-Chiang Frank Wang,
• Abstract summary: We propose Triplane Attention for text-guided 3D generation (TPA3D) TPA3D is an end-to-end trainable GAN-based deep learning model for fast text-to-3D generation. We show that TPA3D generates high-quality 3D textured shapes aligned with fine-grained descriptions.
• Score: 28.33270078863519
• License:
• Abstract: Due to the lack of large-scale text-3D correspondence data, recent text-to-3D generation works mainly rely on utilizing 2D diffusion models for synthesizing 3D data. Since diffusion-based methods typically require significant optimization time for both training and inference, the use of GAN-based models would still be desirable for fast 3D generation. In this work, we propose Triplane Attention for text-guided 3D generation (TPA3D), an end-to-end trainable GAN-based deep learning model for fast text-to-3D generation. With only 3D shape data and their rendered 2D images observed during training, our TPA3D is designed to retrieve detailed visual descriptions for synthesizing the corresponding 3D mesh data. This is achieved by the proposed attention mechanisms on the extracted sentence and word-level text features. In our experiments, we show that TPA3D generates high-quality 3D textured shapes aligned with fine-grained descriptions, while impressive computation efficiency can be observed.

Related papers

• DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data [50.164670363633704]
  We present DIRECT-3D, a diffusion-based 3D generative model for creating high-quality 3D assets from text prompts. Our model is directly trained on extensive noisy and unaligned in-the-wild' 3D assets. We achieve state-of-the-art performance in both single-class generation and text-to-3D generation.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation [96.62867261689037]
  We introduce a novel Visual Prompt-guided text-to-3D diffusion model (VP3D) VP3D explicitly unleashes the visual appearance knowledge in 2D visual prompt to boost text-to-3D generation. Our experiments show that the 2D Visual Prompt in our VP3D significantly eases the learning of visual appearance of 3D models.
  arXiv  Detail & Related papers  (2024-03-25T17:59:31Z)
• LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis [76.43669909525488]
  LATTE3D generates 3D objects in 400ms, and can be further enhanced with fast test-time optimization. We introduce LATTE3D, addressing these limitations to achieve fast, high-quality generation on a significantly larger prompt set.
  arXiv  Detail & Related papers  (2024-03-22T17:59:37Z)
• 3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors [85.11117452560882]
  We present a two-stage text-to-3D generation system, namely 3DTopia, which generates high-quality general 3D assets within 5 minutes using hybrid diffusion priors. The first stage samples from a 3D diffusion prior directly learned from 3D data. Specifically, it is powered by a text-conditioned tri-plane latent diffusion model, which quickly generates coarse 3D samples for fast prototyping. The second stage utilizes 2D diffusion priors to further refine the texture of coarse 3D models from the first stage. The refinement consists of both latent and pixel space optimization for high-quality texture generation
  arXiv  Detail & Related papers  (2024-03-04T17:26:28Z)
• PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion [18.82883336156591]
  We present PI3D, a framework that fully leverages the pre-trained text-to-image diffusion models' ability to generate high-quality 3D shapes from text prompts in minutes. PI3D generates a single 3D shape from text in only 3 minutes and the quality is validated to outperform existing 3D generative models by a large margin.
  arXiv  Detail & Related papers  (2023-12-14T16:04:34Z)
• Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior [52.44678180286886]
  2D diffusion models find a distillation approach that achieves excellent generalization and rich details without any 3D data. We propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity, generalizability, and geometric consistency simultaneously.
  arXiv  Detail & Related papers  (2023-12-11T18:59:18Z)
• Guide3D: Create 3D Avatars from Text and Image Guidance [55.71306021041785]
  Guide3D is a text-and-image-guided generative model for 3D avatar generation based on diffusion models. Our framework produces topologically and structurally correct geometry and high-resolution textures.
  arXiv  Detail & Related papers  (2023-08-18T17:55:47Z)
• Lift3D: Synthesize 3D Training Data by Lifting 2D GAN to 3D Generative Radiance Field [16.15190186574068]
  We propose Lift3D, an inverted 2D-to-3D generation framework to achieve the data generation objectives. By lifting well-disentangled 2D GAN to 3D object NeRF, Lift3D provides explicit 3D information of generated objects. We evaluate the effectiveness of our framework by augmenting autonomous driving datasets.
  arXiv  Detail & Related papers  (2023-04-07T07:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.