TextureDiffusion: Target Prompt Disentangled Editing for Various Texture Transfer

• URL: http://arxiv.org/abs/2409.09610v1
• Date: Sun, 15 Sep 2024 04:34:38 GMT
• Title: TextureDiffusion: Target Prompt Disentangled Editing for Various Texture Transfer
• Authors: Zihan Su, Junhao Zhuang, Chun Yuan,
• Abstract summary: We propose TextureDiffusion, a tuning-free image editing method applied to various texture transfer. query features in self-attention and features in residual blocks are utilized to preserve the structure of the input image. To maintain the background, we introduce an edit localization technique which blends the self-attention results and the intermediate latents.
• Score: 32.53299128227546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, text-guided image editing has achieved significant success. However, existing methods can only apply simple textures like wood or gold when changing the texture of an object. Complex textures such as cloud or fire pose a challenge. This limitation stems from that the target prompt needs to contain both the input image content and <texture>, restricting the texture representation. In this paper, we propose TextureDiffusion, a tuning-free image editing method applied to various texture transfer. Initially, the target prompt is directly set to "<texture>", making the texture disentangled from the input image content to enhance texture representation. Subsequently, query features in self-attention and features in residual blocks are utilized to preserve the structure of the input image. Finally, to maintain the background, we introduce an edit localization technique which blends the self-attention results and the intermediate latents. Comprehensive experiments demonstrate that TextureDiffusion can harmoniously transfer various textures with excellent structure and background preservation.

Related papers

• DragText: Rethinking Text Embedding in Point-based Image Editing [3.1923251959845214]
  We show that during the progressive editing of an input image in a diffusion model, the text embedding remains constant. We propose DragText, which optimize text embedding in conjunction with the dragging process to pair with the modified image embedding.
  arXiv  Detail & Related papers  (2024-07-25T07:57:55Z)
• Infinite Texture: Text-guided High Resolution Diffusion Texture Synthesis [61.189479577198846]
  We present Infinite Texture, a method for generating arbitrarily large texture images from a text prompt. Our approach fine-tunes a diffusion model on a single texture, and learns to embed that statistical distribution in the output domain of the model. At generation time, our fine-tuned diffusion model is used through a score aggregation strategy to generate output texture images of arbitrary resolution on a single GPU.
  arXiv  Detail & Related papers  (2024-05-13T21:53:09Z)
• TexSliders: Diffusion-Based Texture Editing in CLIP Space [17.449209402077276]
  We analyze existing editing methods and show that they are not directly applicable to textures. We propose a novel approach that instead manipulates CLIP image embeddings to condition the diffusion generation.
  arXiv  Detail & Related papers  (2024-05-01T17:57:21Z)
• Compositional Neural Textures [25.885557234297835]
  This work introduces a fully unsupervised approach for representing textures using a compositional neural model. We represent each texton as a 2D Gaussian function whose spatial support approximates its shape, and an associated feature that encodes its detailed appearance. This approach enables a wide range of applications, including transferring appearance from an image texture to another image, diversifying textures, revealing/modifying texture variations, edit propagation, texture animation, and direct texton manipulation.
  arXiv  Detail & Related papers  (2024-04-18T21:09:34Z)
• TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion [64.49276500129092]
  TextureDreamer is an image-guided texture synthesis method. It can transfer relightable textures from a small number of input images to target 3D shapes across arbitrary categories.
  arXiv  Detail & Related papers  (2024-01-17T18:55:49Z)
• Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Details [12.660352353074012]
  We propose Text2Scene, a method to automatically create realistic textures for virtual scenes composed of multiple objects. Our pipeline adds detailed texture on labeled 3D geometries in the room such that the generated colors respect the hierarchical structure or semantic parts that are often composed of similar materials.
  arXiv  Detail & Related papers  (2023-08-31T17:37:23Z)
• TEXTure: Text-Guided Texturing of 3D Shapes [71.13116133846084]
  We present TEXTure, a novel method for text-guided editing, editing, and transfer of textures for 3D shapes. We define a trimap partitioning process that generates seamless 3D textures without requiring explicit surface textures.
  arXiv  Detail & Related papers  (2023-02-03T13:18:45Z)
• Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting [53.708523312636096]
  We present Imagen Editor, a cascaded diffusion model built, by fine-tuning on text-guided image inpainting. edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training. To improve qualitative and quantitative evaluation, we introduce EditBench, a systematic benchmark for text-guided image inpainting.
  arXiv  Detail & Related papers  (2022-12-13T21:25:11Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)
• Image Inpainting Guided by Coherence Priors of Semantics and Textures [62.92586889409379]
  We introduce coherence priors between the semantics and textures which make it possible to concentrate on completing separate textures in a semantic-wise manner. We also propose two coherence losses to constrain the consistency between the semantics and the inpainted image in terms of the overall structure and detailed textures.
  arXiv  Detail & Related papers  (2020-12-15T02:59:37Z)
• Texture Transform Attention for Realistic Image Inpainting [6.275013056564918]
  We propose a Texture Transform Attention network that better produces the missing region inpainting with fine details. Texture Transform Attention is used to create a new reassembled texture map using fine textures and coarse semantics. We evaluate our model end-to-end with the publicly available datasets CelebA-HQ and Places2.
  arXiv  Detail & Related papers  (2020-12-08T06:28:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.