Directional Texture Editing for 3D Models

• URL: http://arxiv.org/abs/2309.14872v4
• Date: Wed, 6 Mar 2024 11:07:50 GMT
• Title: Directional Texture Editing for 3D Models
• Authors: Shengqi Liu, Zhuo Chen, Jingnan Gao, Yichao Yan, Wenhan Zhu, Jiangjing Lyu, Xiaokang Yang
• Abstract summary: ITEM3D is designed for automatic textbf3D object editing according to the text textbfInstructions. Leveraging the diffusion models and the differentiable rendering, ITEM3D takes the rendered images as the bridge of text and 3D representation.
• Score: 51.31499400557996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Texture editing is a crucial task in 3D modeling that allows users to automatically manipulate the surface materials of 3D models. However, the inherent complexity of 3D models and the ambiguous text description lead to the challenge in this task. To address this challenge, we propose ITEM3D, a \textbf{T}exture \textbf{E}diting \textbf{M}odel designed for automatic \textbf{3D} object editing according to the text \textbf{I}nstructions. Leveraging the diffusion models and the differentiable rendering, ITEM3D takes the rendered images as the bridge of text and 3D representation, and further optimizes the disentangled texture and environment map. Previous methods adopted the absolute editing direction namely score distillation sampling (SDS) as the optimization objective, which unfortunately results in the noisy appearance and text inconsistency. To solve the problem caused by the ambiguous text, we introduce a relative editing direction, an optimization objective defined by the noise difference between the source and target texts, to release the semantic ambiguity between the texts and images. Additionally, we gradually adjust the direction during optimization to further address the unexpected deviation in the texture domain. Qualitative and quantitative experiments show that our ITEM3D outperforms the state-of-the-art methods on various 3D objects. We also perform text-guided relighting to show explicit control over lighting. Our project page: https://shengqiliu1.github.io/ITEM3D.

Related papers

• ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models [65.22994156658918]
  We present a method that learns to generate multi-view images in a single denoising process from real-world data. We design an autoregressive generation that renders more 3D-consistent images at any viewpoint.
  arXiv  Detail & Related papers  (2024-03-04T07:57:05Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• Learning Naturally Aggregated Appearance for Efficient 3D Editing [94.47518916521065]
  We propose to replace the color field with an explicit 2D appearance aggregation, also called canonical image. To avoid the distortion effect and facilitate convenient editing, we complement the canonical image with a projection field that maps 3D points onto 2D pixels for texture lookup. Our representation, dubbed AGAP, well supports various ways of 3D editing (e.g., stylization, interactive drawing, and content extraction) with no need of re-optimization.
  arXiv  Detail & Related papers  (2023-12-11T18:59:31Z)
• ATT3D: Amortized Text-to-3D Object Synthesis [78.96673650638365]
  We amortize optimization over text prompts by training on many prompts simultaneously with a unified model, instead of separately. Our framework - Amortized text-to-3D (ATT3D) - enables knowledge-sharing between prompts to generalize to unseen setups and smooths between text for novel assets and simple animations.
  arXiv  Detail & Related papers  (2023-06-06T17:59:10Z)
• TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision [114.56048848216254]
  We present a novel framework, TAPS3D, to train a text-guided 3D shape generator with pseudo captions. Based on rendered 2D images, we retrieve relevant words from the CLIP vocabulary and construct pseudo captions using templates. Our constructed captions provide high-level semantic supervision for generated 3D shapes.
  arXiv  Detail & Related papers  (2023-03-23T13:53:16Z)
• Vox-E: Text-guided Voxel Editing of 3D Objects [14.88446525549421]
  Large scale text-guided diffusion models have garnered significant attention due to their ability to synthesize diverse images. We present a technique that harnesses the power of latent diffusion models for editing existing 3D objects.
  arXiv  Detail & Related papers  (2023-03-21T17:36:36Z)
• Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models [44.34479731617561]
  We introduce explicit 3D shape priors into the CLIP-guided 3D optimization process. We present a simple yet effective approach that directly bridges the text and image modalities with a powerful text-to-image diffusion model. Our method, Dream3D, is capable of generating imaginative 3D content with superior visual quality and shape accuracy.
  arXiv  Detail & Related papers  (2022-12-28T18:23:47Z)
• TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition [39.312567993736025]
  We propose TANGO, which transfers the appearance style of a given 3D shape according to a text prompt in a photorealistic manner. We show that TANGO outperforms existing methods of text-driven 3D style transfer in terms of photorealistic quality, consistency of 3D geometry, and robustness when stylizing low-quality meshes.
  arXiv  Detail & Related papers  (2022-10-20T13:52:18Z)
• Texturify: Generating Textures on 3D Shape Surfaces [34.726179801982646]
  We propose Texturify to learn a 3D shape that predicts texture on the 3D input. Our method does not require any 3D color supervision to learn 3D objects.
  arXiv  Detail & Related papers  (2022-04-05T18:00:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.