Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation

• URL: http://arxiv.org/abs/2405.06948v1
• Date: Sat, 11 May 2024 08:11:25 GMT
• Title: Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation
• Authors: Shengyuan Liu, Bo Wang, Ye Ma, Te Yang, Xipeng Cao, Quan Chen, Han Li, Di Dong, Peng Jiang,
• Abstract summary: We propose a subject-driven generation framework and introduce training-free guidance to intervene in the generative process during inference time. Notably, our method exhibits exceptional zero-shot generation ability, especially in the challenging task of compositional generation.
• Score: 22.949365270116335
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing subject-driven text-to-image generation models suffer from tedious fine-tuning steps and struggle to maintain both text-image alignment and subject fidelity. For generating compositional subjects, it often encounters problems such as object missing and attribute mixing, where some subjects in the input prompt are not generated or their attributes are incorrectly combined. To address these limitations, we propose a subject-driven generation framework and introduce training-free guidance to intervene in the generative process during inference time. This approach strengthens the attention map, allowing for precise attribute binding and feature injection for each subject. Notably, our method exhibits exceptional zero-shot generation ability, especially in the challenging task of compositional generation. Furthermore, we propose a novel metric GroundingScore to evaluate subject alignment thoroughly. The obtained quantitative results serve as compelling evidence showcasing the effectiveness of our proposed method. The code will be released soon.

Related papers

• MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance [6.4680449907623006]
  This research introduces the MS-Diffusion framework for layout-guided zero-shot image personalization with multi-subjects. The proposed multi-subject cross-attention orchestrates inter-subject compositions while preserving the control of texts.
  arXiv  Detail & Related papers  (2024-06-11T12:32:53Z)
• Information Theoretic Text-to-Image Alignment [49.396917351264655]
  We present a novel method that relies on an information-theoretic alignment measure to steer image generation. Our method is on-par or superior to the state-of-the-art, yet requires nothing but a pre-trained denoising network to estimate MI.
  arXiv  Detail & Related papers  (2024-05-31T12:20:02Z)
• Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation [60.943159830780154]
  We introduce Bounded Attention, a training-free method for bounding the information flow in the sampling process. We demonstrate that our method empowers the generation of multiple subjects that better align with given prompts and layouts.
  arXiv  Detail & Related papers  (2024-03-25T17:52:07Z)
• Selectively Informative Description can Reduce Undesired Embedding Entanglements in Text-to-Image Personalization [5.141049647900161]
  We propose SID(Selectively Informative Description), a text description strategy that deviates from the prevalent approach of only characterizing the subject's class identification. We present comprehensive experimental results along with analyses of cross-attention maps, subject-alignment, non-subject-disentanglement, and text-alignment.
  arXiv  Detail & Related papers  (2024-03-22T16:35:38Z)
• Harmonizing Visual and Textual Embeddings for Zero-Shot Text-to-Image Customization [23.04290567321589]
  A surge of text-to-image (T2I) models and their customization methods generate new images of a user-provided subject. These zero-shot customization methods encode the image of a specified subject into a visual embedding which is then utilized alongside the textual embedding for diffusion guidance. We propose visual embedding which effectively harmonizes with the given textual embedding. We also adopt the visual-only embedding and inject the subject's clear features utilizing a self-attention swap.
  arXiv  Detail & Related papers  (2024-03-21T06:03:51Z)
• Training-Free Consistent Text-to-Image Generation [80.4814768762066]
  Text-to-image models can portray the same subject across diverse prompts. Existing approaches fine-tune the model to teach it new words that describe specific user-provided subjects. We present ConsiStory, a training-free approach that enables consistent subject generation by sharing the internal activations of the pretrained model.
  arXiv  Detail & Related papers  (2024-02-05T18:42:34Z)
• Instilling Multi-round Thinking to Text-guided Image Generation [72.2032630115201]
  Single-round generation often overlooks crucial details, particularly in the realm of fine-grained changes like shoes or sleeves. We introduce a new self-supervised regularization, ie, multi-round regularization, which is compatible with existing methods. It builds upon the observation that the modification order generally should not affect the final result.
  arXiv  Detail & Related papers  (2024-01-16T16:19:58Z)
• DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation [50.39533637201273]
  We propose DisenBooth, an identity-preserving disentangled tuning framework for subject-driven text-to-image generation. By combining the identity-preserved embedding and identity-irrelevant embedding, DisenBooth demonstrates more generation flexibility and controllability.
  arXiv  Detail & Related papers  (2023-05-05T09:08:25Z)
• Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models [103.61066310897928]
  Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. We introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness
  arXiv  Detail & Related papers  (2023-01-31T18:10:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.