InTeX: Interactive Text-to-texture Synthesis via Unified Depth-aware Inpainting

• URL: http://arxiv.org/abs/2403.11878v1
• Date: Mon, 18 Mar 2024 15:31:57 GMT
• Title: InTeX: Interactive Text-to-texture Synthesis via Unified Depth-aware Inpainting
• Authors: Jiaxiang Tang, Ruijie Lu, Xiaokang Chen, Xiang Wen, Gang Zeng, Ziwei Liu,
• Abstract summary: We introduce InteX, a novel framework for interactive text-to-texture synthesis. InteX includes a user-friendly interface that facilitates interaction and control throughout the synthesis process. We develop a depth-aware inpainting model that integrates depth information with inpainting cues, effectively mitigating 3D inconsistencies.
• Score: 46.330305910974246
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-texture synthesis has become a new frontier in 3D content creation thanks to the recent advances in text-to-image models. Existing methods primarily adopt a combination of pretrained depth-aware diffusion and inpainting models, yet they exhibit shortcomings such as 3D inconsistency and limited controllability. To address these challenges, we introduce InteX, a novel framework for interactive text-to-texture synthesis. 1) InteX includes a user-friendly interface that facilitates interaction and control throughout the synthesis process, enabling region-specific repainting and precise texture editing. 2) Additionally, we develop a unified depth-aware inpainting model that integrates depth information with inpainting cues, effectively mitigating 3D inconsistencies and improving generation speed. Through extensive experiments, our framework has proven to be both practical and effective in text-to-texture synthesis, paving the way for high-quality 3D content creation.

Related papers

• GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation [35.04723374116026]
  Large-scale text-to-image (T2I) models have shown astonishing results in text-to-image (T2I) generation. Applying these models to synthesize textures for 3D geometries remains challenging due to the domain gap between 2D images and textures on a 3D surface. We propose a novel text-to-texture synthesis framework that leverages pretrained diffusion models.
  arXiv  Detail & Related papers  (2024-09-27T02:32:42Z)
• Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects [54.80813150893719]
  We introduce Meta 3D TextureGen: a new feedforward method comprised of two sequential networks aimed at generating high-quality textures in less than 20 seconds. Our method state-of-the-art results in quality and speed by conditioning a text-to-image model on 3D semantics in 2D space and fusing them into a complete and high-resolution UV texture map. In addition, we introduce a texture enhancement network that is capable of up-scaling any texture by an arbitrary ratio, producing 4k pixel resolution textures.
  arXiv  Detail & Related papers  (2024-07-02T17:04:34Z)
• WordRobe: Text-Guided Generation of Textured 3D Garments [30.614451083408266]
  "WordRobe" is a novel framework for the generation of unposed & textured 3D garment meshes from user-friendly text prompts. We demonstrate superior performance over current SOTAs for learning 3D garment latent space, garment synthesis, and text-driven texture synthesis.
  arXiv  Detail & Related papers  (2024-03-26T09:44:34Z)
• Text-Guided 3D Face Synthesis -- From Generation to Editing [53.86765812392627]
  We propose a unified text-guided framework from face generation to editing. We employ a fine-tuned texture diffusion model to enhance texture quality in both RGB and YUV space. We propose a self-guided consistency weight strategy to improve editing efficacy while preserving consistency.
  arXiv  Detail & Related papers  (2023-12-01T06:36:23Z)
• TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models [77.85129451435704]
  We present a new method to synthesize textures for 3D, using large-scale-guided image diffusion models. Specifically, we leverage latent diffusion models, apply the set denoising model and aggregate denoising text map.
  arXiv  Detail & Related papers  (2023-10-20T19:15:29Z)
• ATT3D: Amortized Text-to-3D Object Synthesis [78.96673650638365]
  We amortize optimization over text prompts by training on many prompts simultaneously with a unified model, instead of separately. Our framework - Amortized text-to-3D (ATT3D) - enables knowledge-sharing between prompts to generalize to unseen setups and smooths between text for novel assets and simple animations.
  arXiv  Detail & Related papers  (2023-06-06T17:59:10Z)
• Text2Tex: Text-driven Texture Synthesis via Diffusion Models [31.773823357617093]
  We present Text2Tex, a novel method for generating high-quality textures for 3D meshes from text prompts. Our method incorporates inpainting into a pre-trained depth-aware image diffusion model to progressively synthesize high resolution partial textures from multiple viewpoints.
  arXiv  Detail & Related papers  (2023-03-20T19:02:13Z)
• TEXTure: Text-Guided Texturing of 3D Shapes [71.13116133846084]
  We present TEXTure, a novel method for text-guided editing, editing, and transfer of textures for 3D shapes. We define a trimap partitioning process that generates seamless 3D textures without requiring explicit surface textures.
  arXiv  Detail & Related papers  (2023-02-03T13:18:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.