Towards Implicit Text-Guided 3D Shape Generation

• URL: http://arxiv.org/abs/2203.14622v1
• Date: Mon, 28 Mar 2022 10:20:03 GMT
• Title: Towards Implicit Text-Guided 3D Shape Generation
• Authors: Zhengzhe Liu, Yi Wang, Xiaojuan Qi, Chi-Wing Fu
• Abstract summary: This work explores the challenging task of generating 3D shapes from text. We propose a new approach for text-guided 3D shape generation, capable of producing high-fidelity shapes with colors that match the given text description.
• Score: 81.22491096132507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we explore the challenging task of generating 3D shapes from text. Beyond the existing works, we propose a new approach for text-guided 3D shape generation, capable of producing high-fidelity shapes with colors that match the given text description. This work has several technical contributions. First, we decouple the shape and color predictions for learning features in both texts and shapes, and propose the word-level spatial transformer to correlate word features from text with spatial features from shape. Also, we design a cyclic loss to encourage consistency between text and shape, and introduce the shape IMLE to diversify the generated shapes. Further, we extend the framework to enable text-guided shape manipulation. Extensive experiments on the largest existing text-shape benchmark manifest the superiority of this work. The code and the models are available at https://github.com/liuzhengzhe/Towards-Implicit Text-Guided-Shape-Generation.

Related papers

• HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generation [55.95329424826433]
  We propose HyperSDFusion, a dual-branch diffusion model that generates 3D shapes from a given text. We learn the hierarchical representations of text and 3D shapes in hyperbolic space. Our method is the first to explore the hyperbolic hierarchical representation for text-to-shape generation.
  arXiv  Detail & Related papers  (2024-03-01T08:57:28Z)
• EXIM: A Hybrid Explicit-Implicit Representation for Text-Guided 3D Shape Generation [124.27302003578903]
  This paper presents a new text-guided technique for generating 3D shapes. We leverage a hybrid 3D representation, namely EXIM, combining the strengths of explicit and implicit representations. We demonstrate the applicability of our approach to generate indoor scenes with consistent styles using text-induced 3D shapes.
  arXiv  Detail & Related papers  (2023-11-03T05:01:51Z)
• DreamStone: Image as Stepping Stone for Text-Guided 3D Shape Generation [105.97545053660619]
  We present a new text-guided 3D shape generation approach DreamStone. It uses images as a stepping stone to bridge the gap between text and shape modalities for generating 3D shapes without requiring paired text and 3D data. Our approach is generic, flexible, and scalable, and it can be easily integrated with various SVR models to expand the generative space and improve the generative fidelity.
  arXiv  Detail & Related papers  (2023-03-24T03:56:23Z)
• TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision [114.56048848216254]
  We present a novel framework, TAPS3D, to train a text-guided 3D shape generator with pseudo captions. Based on rendered 2D images, we retrieve relevant words from the CLIP vocabulary and construct pseudo captions using templates. Our constructed captions provide high-level semantic supervision for generated 3D shapes.
  arXiv  Detail & Related papers  (2023-03-23T13:53:16Z)
• ISS: Image as Stetting Stone for Text-Guided 3D Shape Generation [91.37036638939622]
  This paper presents a new framework called Image as Stepping Stone (ISS) for the task by introducing 2D image as a stepping stone to connect the two modalities. Our key contribution is a two-stage feature-space-alignment approach that maps CLIP features to shapes. We formulate a text-guided shape stylization module to dress up the output shapes with novel textures.
  arXiv  Detail & Related papers  (2022-09-09T06:54:21Z)
• ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model [16.431391515731367]
  Existing methods to generate text-conditioned 3D shapes consume an entire text prompt to generate a 3D shape in a single step. We introduce a method to generate a 3D shape distribution conditioned on an initial phrase, that gradually evolves as more phrases are added. Results show that our method can generate shapes consistent with text descriptions, and shapes evolve gradually as more phrases are added.
  arXiv  Detail & Related papers  (2022-07-19T17:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.