Scribble-Guided Diffusion for Training-free Text-to-Image Generation

• URL: http://arxiv.org/abs/2409.08026v1
• Date: Thu, 12 Sep 2024 13:13:07 GMT
• Title: Scribble-Guided Diffusion for Training-free Text-to-Image Generation
• Authors: Seonho Lee, Jiho Choi, Seohyun Lim, Jiwook Kim, Hyunjung Shim,
• Abstract summary: Scribble-Guided Diffusion (ScribbleDiff) is a training-free approach that utilizes simple user-provided scribbles as visual prompts to guide image generation. We introduce moment alignment and scribble propagation, which allow for more effective and flexible alignment between generated images and scribble inputs.
• Score: 17.930032337081673
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent advancements in text-to-image diffusion models have demonstrated remarkable success, yet they often struggle to fully capture the user's intent. Existing approaches using textual inputs combined with bounding boxes or region masks fall short in providing precise spatial guidance, often leading to misaligned or unintended object orientation. To address these limitations, we propose Scribble-Guided Diffusion (ScribbleDiff), a training-free approach that utilizes simple user-provided scribbles as visual prompts to guide image generation. However, incorporating scribbles into diffusion models presents challenges due to their sparse and thin nature, making it difficult to ensure accurate orientation alignment. To overcome these challenges, we introduce moment alignment and scribble propagation, which allow for more effective and flexible alignment between generated images and scribble inputs. Experimental results on the PASCAL-Scribble dataset demonstrate significant improvements in spatial control and consistency, showcasing the effectiveness of scribble-based guidance in diffusion models. Our code is available at https://github.com/kaist-cvml-lab/scribble-diffusion.

Related papers

• Training-free Composite Scene Generation for Layout-to-Image Synthesis [29.186425845897947]
  This paper introduces a novel training-free approach designed to overcome adversarial semantic intersections during the diffusion conditioning phase. We propose two innovative constraints: 1) an inter-token constraint that resolves token conflicts to ensure accurate concept synthesis; and 2) a self-attention constraint that improves pixel-to-pixel relationships. Our evaluations confirm the effectiveness of leveraging layout information for guiding the diffusion process, generating content-rich images with enhanced fidelity and complexity.
  arXiv  Detail & Related papers  (2024-07-18T15:48:07Z)
• Coherent Zero-Shot Visual Instruction Generation [15.0521272616551]
  This paper introduces a simple, training-free framework to tackle the issues of generating visual instructions. Our approach systematically integrates text comprehension and image generation to ensure visual instructions are visually appealing. Our experiments show that our approach can visualize coherent and visually pleasing instructions.
  arXiv  Detail & Related papers  (2024-06-06T17:59:44Z)
• Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [65.7968515029306]
  We propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS) A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt.
  arXiv  Detail & Related papers  (2024-02-28T06:07:07Z)
• Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering [118.53208190209517]
  We propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. Our method can effectively learn the prompts to improve the matches between the input text and the generated images.
  arXiv  Detail & Related papers  (2024-01-12T03:46:29Z)
• Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models [46.18013380882767]
  This work focuses on inverting the diffusion model to obtain interpretable language prompts directly. We leverage the findings that different timesteps of the diffusion process cater to different levels of detail in an image. We show that our approach can identify semantically interpretable and meaningful prompts for a target image.
  arXiv  Detail & Related papers  (2023-12-19T18:47:30Z)
• MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask [84.84034179136458]
  A crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning. We propose an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models.
  arXiv  Detail & Related papers  (2023-09-08T15:53:37Z)
• Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation [1.0152838128195465]
  We propose a method for spatially controlling text-to-image generation without further training of diffusion models. Our aim is to control the attention maps according to given semantic masks and text prompts.
  arXiv  Detail & Related papers  (2023-08-11T09:15:22Z)
• LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [121.45667242282721]
  We propose a coarse-to-fine paradigm to achieve layout planning and image generation. Our proposed method outperforms the state-of-the-art models in terms of photorealistic layout and image generation.
  arXiv  Detail & Related papers  (2023-08-09T17:45:04Z)
• Zero-shot spatial layout conditioning for text-to-image diffusion models [52.24744018240424]
  Large-scale text-to-image diffusion models have significantly improved the state of the art in generative image modelling. We consider image generation from text associated with segments on the image canvas, which combines an intuitive natural language interface with precise spatial control over the generated content. We propose ZestGuide, a zero-shot segmentation guidance approach that can be plugged into pre-trained text-to-image diffusion models.
  arXiv  Detail & Related papers  (2023-06-23T19:24:48Z)
• Self-Guided Diffusion Models [53.825634944114285]
  We propose a framework for self-guided diffusion models. Our method provides guidance signals at various image granularities. Our experiments on single-label and multi-label image datasets demonstrate that self-labeled guidance always outperforms diffusion models without guidance.
  arXiv  Detail & Related papers  (2022-10-12T17:57:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.