Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects

• URL: http://arxiv.org/abs/2411.18936v2
• Date: Mon, 24 Mar 2025 19:58:03 GMT
• Title: Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects
• Abstract summary: Self-Cross Diffusion Guidance penalizes the overlap between cross-attention maps and the aggregated self-attention map.<n>For each subject, we aggregate self-attention maps of patches with higher cross-attention values.<n>Our method boosts the performance of both Unet-based and Transformer-based diffusion models.
• Score: 1.419173007355272
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Diffusion models achieved unprecedented fidelity and diversity for synthesizing image, video, 3D assets, etc. However, subject mixing is an unresolved issue for diffusion-based image synthesis, particularly for synthesizing multiple similar-looking subjects. We propose Self-Cross Diffusion Guidance to penalize the overlap between cross-attention maps and the aggregated self-attention map. Compared to previous methods based on self-attention or cross-attention alone, our guidance is more effective in eliminating subject mixing. What's more, our guidance addresses subject mixing for all relevant patches beyond the most discriminant one, e.g., the beak of a bird. For each subject, we aggregate self-attention maps of patches with higher cross-attention values. Thus, the aggregated self-attention map forms a region that the whole subject attends to. Our training-free method boosts the performance of both Unet-based and Transformer-based diffusion models such as the Stable Diffusion series. We also release a similar subjects dataset (SSD), a challenging benchmark, and utilize GPT-4o for automatic and reliable evaluation. Extensive qualitative and quantitative results demonstrate the effectiveness of our self-cross diffusion guidance.

Related papers

• AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesis [12.76456980137364]
  We propose an aggregation-and-Isolating cross-attention approach to diffusion models for Text-to-Image synthesis.<n>Our AI-T2I exhibits excellent generalization across other tasks, e.g., controllable layout generation and personalized generation.
  arXiv  Detail & Related papers  (2026-05-25T12:17:32Z)
• Authentic Discrete Diffusion Model [72.31371542619121]
  Authentic Discrete Diffusion (ADD) framework redefines prior pseudo-discrete approaches.<n>ADD reformulates the diffusion input by directly using float-encoded one-hot class data.<n> experiments demonstrate that ADD achieves superior performance on classification tasks compared to the baseline.
  arXiv  Detail & Related papers  (2025-10-01T15:51:10Z)
• UniSegDiff: Boosting Unified Lesion Segmentation via a Staged Diffusion Model [53.34835793648352]
  We propose UniSegDiff, a novel diffusion model framework for lesion segmentation.<n>UniSegDiff addresses lesion segmentation in a unified manner across multiple modalities and organs.<n> Comprehensive experimental results demonstrate that UniSegDiff significantly outperforms previous state-of-the-art (SOTA) approaches.
  arXiv  Detail & Related papers  (2025-07-24T12:33:10Z)
• Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflection [28.82743020243849]
  Existing text-to-image diffusion models often fail to maintain high image quality and high prompt-image alignment for challenging prompts. We propose diffusion self-reflection that alternately performs denoising and inversion. We derive Zigzag Diffusion Sampling (Z-Sampling), a novel self-reflection-based diffusion sampling method.
  arXiv  Detail & Related papers  (2024-12-14T16:42:41Z)
• Inversion Circle Interpolation: Diffusion-based Image Augmentation for Data-scarce Classification [6.442738337380714]
  We argue that diffusion-based DA methods cannot take account of both faithfulness and diversity, which are two critical keys for generating high-quality samples. We propose a novel Diffusion-based DA method: Diff-II.
  arXiv  Detail & Related papers  (2024-08-29T05:05:02Z)
• Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas [33.334956022229846]
  We propose the Merge-Attend-Diffuse operator, which can be plugged into different types of pretrained diffusion models used in a joint diffusion setting. Specifically, we merge the diffusion paths, reprogramming self- and cross-attention to operate on the aggregated latent space. Our method maintains compatibility with the input prompt and visual quality of the generated images while increasing their semantic coherence.
  arXiv  Detail & Related papers  (2024-08-28T09:22:32Z)
• Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation [60.943159830780154]
  We introduce Bounded Attention, a training-free method for bounding the information flow in the sampling process. We demonstrate that our method empowers the generation of multiple subjects that better align with given prompts and layouts.
  arXiv  Detail & Related papers  (2024-03-25T17:52:07Z)
• SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
  We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks. To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
  arXiv  Detail & Related papers  (2024-03-25T10:30:22Z)
• Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models [23.786473791344395]
  Cross-attention layers in diffusion models tend to disproportionately focus on certain tokens during the generation process. We introduce attention regulation, an on-the-fly optimization approach at inference time to align attention maps with the input text prompt. Experiment results show that our method consistently outperforms other baselines.
  arXiv  Detail & Related papers  (2024-03-11T02:18:27Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• Multilevel Saliency-Guided Self-Supervised Learning for Image Anomaly Detection [15.212031255539022]
  Anomaly detection (AD) is a fundamental task in computer vision. We propose CutSwap, which leverages saliency guidance to incorporate semantic cues for augmentation. CutSwap achieves state-of-the-art AD performance on two mainstream AD benchmark datasets.
  arXiv  Detail & Related papers  (2023-11-30T08:03:53Z)
• Reverse Stable Diffusion: What prompt was used to generate this image? [73.10116197883303]
  We study the task of predicting the prompt embedding given an image generated by a generative diffusion model. We propose a novel learning framework comprising a joint prompt regression and multi-label vocabulary classification objective. We conduct experiments on the DiffusionDB data set, predicting text prompts from images generated by Stable Diffusion.
  arXiv  Detail & Related papers  (2023-08-02T23:39:29Z)
• Your Diffusion Model is Secretly a Zero-Shot Classifier [90.40799216880342]
  We show that density estimates from large-scale text-to-image diffusion models can be leveraged to perform zero-shot classification. Our generative approach to classification attains strong results on a variety of benchmarks. Our results are a step toward using generative over discriminative models for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T17:59:56Z)
• Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models [103.61066310897928]
  Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect, where the model fails to generate one or more of the subjects from the input prompt. We introduce the concept of Generative Semantic Nursing (GSN), where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness
  arXiv  Detail & Related papers  (2023-01-31T18:10:38Z)
• Person Image Synthesis via Denoising Diffusion Model [116.34633988927429]
  We show how denoising diffusion models can be applied for high-fidelity person image synthesis. Our results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios.
  arXiv  Detail & Related papers  (2022-11-22T18:59:50Z)
• Self-Guided Diffusion Models [53.825634944114285]
  We propose a framework for self-guided diffusion models. Our method provides guidance signals at various image granularities. Our experiments on single-label and multi-label image datasets demonstrate that self-labeled guidance always outperforms diffusion models without guidance.
  arXiv  Detail & Related papers  (2022-10-12T17:57:58Z)
• Improving Sample Quality of Diffusion Models Using Self-Attention Guidance [36.42984435784378]
  Self-Attention Guidance (SAG) improves the performance of various diffusion models. SAG adversarially blurs only the regions that diffusion models attend to at each and guides them accordingly. Our results show that our SAG improves the performance of various diffusion models, including ADM, IDDPM, Stable Diffusion, and DiT.
  arXiv  Detail & Related papers  (2022-10-03T13:50:58Z)
• Light Field Saliency Detection with Dual Local Graph Learning andReciprocative Guidance [148.9832328803202]
  We model the infor-mation fusion within focal stack via graph networks. We build a novel dual graph modelto guide the focal stack fusion process using all-focus pat-terns.
  arXiv  Detail & Related papers  (2021-10-02T00:54:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.