Text to Mesh Without 3D Supervision Using Limit Subdivision

• URL: http://arxiv.org/abs/2203.13333v1
• Date: Thu, 24 Mar 2022 20:36:28 GMT
• Title: Text to Mesh Without 3D Supervision Using Limit Subdivision
• Authors: Nasir Khalid, Tianhao Xie, Eugene Belilovsky, Tiberiu Popa
• Abstract summary: We present a technique for zero-shot generation of a 3D model using only a target text prompt. We rely on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model.
• Score: 13.358081015190255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without a generative model or any 3D supervision our method deforms a control shape of a limit subdivided surface along with a texture map and normal map to obtain a 3D model asset that matches the input text prompt and can be deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimization on mesh parameters directly to generate shape and texture. To improve the quality of results we also introduce a set of techniques such as render augmentations, primitive selection, prompt augmentation that guide the mesh towards a suitable result.

Related papers

• Garment3DGen: 3D Garment Stylization and Texture Generation [11.836357439129301]
  Garment3DGen is a new method to synthesize 3D garment assets from a base mesh given a single input image as guidance. The generated assets can be directly draped and simulated on human bodies.
  arXiv  Detail & Related papers  (2024-03-27T17:59:33Z)
• Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives [70.32817882783608]
  We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
  arXiv  Detail & Related papers  (2023-07-11T17:58:31Z)
• DreamStone: Image as Stepping Stone for Text-Guided 3D Shape Generation [105.97545053660619]
  We present a new text-guided 3D shape generation approach DreamStone. It uses images as a stepping stone to bridge the gap between text and shape modalities for generating 3D shapes without requiring paired text and 3D data. Our approach is generic, flexible, and scalable, and it can be easily integrated with various SVR models to expand the generative space and improve the generative fidelity.
  arXiv  Detail & Related papers  (2023-03-24T03:56:23Z)
• TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision [114.56048848216254]
  We present a novel framework, TAPS3D, to train a text-guided 3D shape generator with pseudo captions. Based on rendered 2D images, we retrieve relevant words from the CLIP vocabulary and construct pseudo captions using templates. Our constructed captions provide high-level semantic supervision for generated 3D shapes.
  arXiv  Detail & Related papers  (2023-03-23T13:53:16Z)
• MeshDiffusion: Score-based Generative 3D Mesh Modeling [68.40770889259143]
  We consider the task of generating realistic 3D shapes for automatic scene generation and physical simulation. We take advantage of the graph structure of meshes and use a simple yet very effective generative modeling method to generate 3D meshes. Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parametrization.
  arXiv  Detail & Related papers  (2023-03-14T17:59:01Z)
• Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models [44.34479731617561]
  We introduce explicit 3D shape priors into the CLIP-guided 3D optimization process. We present a simple yet effective approach that directly bridges the text and image modalities with a powerful text-to-image diffusion model. Our method, Dream3D, is capable of generating imaginative 3D content with superior visual quality and shape accuracy.
  arXiv  Detail & Related papers  (2022-12-28T18:23:47Z)
• ClipFace: Text-guided Editing of Textured 3D Morphable Models [33.83015491013442]
  We propose ClipFace, a novel self-supervised approach for text-guided editing of textured 3D morphable model of faces. We employ user-friendly language prompts to enable control of the expressions as well as appearance of 3D faces. Our model is trained in a self-supervised fashion by exploiting differentiable rendering and losses based on a pre-trained CLIP model.
  arXiv  Detail & Related papers  (2022-12-02T19:01:08Z)
• 3D-TOGO: Towards Text-Guided Cross-Category 3D Object Generation [107.46972849241168]
  3D-TOGO model generates 3D objects in the form of the neural radiance field with good texture. Experiments on the largest 3D object dataset (i.e., ABO) are conducted to verify that 3D-TOGO can better generate high-quality 3D objects.
  arXiv  Detail & Related papers  (2022-12-02T11:31:49Z)
• Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images [82.32776379815712]
  We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses. We adopt to further improve the shape quality by leveraging cross-view information with a graph convolution network. Our model is robust to the quality of the initial mesh and the error of camera pose, and can be combined with a differentiable function for test-time optimization.
  arXiv  Detail & Related papers  (2022-04-21T03:42:31Z)
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction [0.0]
  Learning-based approaches for 3D model reconstruction have attracted attention owing to its modern applications. We present a novel sampling algorithm by optimizing the gradient of predicted coordinates based on the variance on the sampling image. We also adopt Frechet Inception Distance (FID) to form a loss function in learning, which helps bridging the gap between rendered images and input images.
  arXiv  Detail & Related papers  (2021-04-29T07:52:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.