AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesis

• URL: http://arxiv.org/abs/2605.25763v3
• Date: Wed, 27 May 2026 11:23:41 GMT
• Title: AI-T2I: Aggregating-and-Isolating Cross-Attention to Diffusion Models for Text-to-Image Synthesis
• Abstract summary: We propose an aggregation-and-Isolating cross-attention approach to diffusion models for Text-to-Image synthesis.<n>Our AI-T2I exhibits excellent generalization across other tasks, e.g., controllable layout generation and personalized generation.
• Score: 12.76456980137364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image synthesis has made significant progress, benefiting from the strong generative capabilities of diffusion models. However, these models struggle to achieve precise text-to-image alignment within cross-attention maps during the denoising process. Existing works primarily focus on inter-subject-token activations (i.e., cross-attention scores) overlap for different subjects, overlooking the intra-subject-token activations scattering issue for identical subjects. In this paper, we propose an Aggregating-and-Isolating cross-attention approach to diffusion models for Text-to-Image synthesis, dubbed AI-T2I. Technically, to address the scattering issue, we devise an aggregation loss to identify and consolidate the scattered intra-token activations, which implicitly helps mitigate the potential overlap issue. Upon that, an isolation loss is further introduced to push the inter-token activations apart, thus fulfilling precise text-to-image alignment. Extensive experiments on various benchmarks demonstrate the superiority of AI-T2I over the state-of-the-art works for text-to-image synthesis. Furthermore, our AI-T2I exhibits excellent generalization across other tasks, e.g., controllable layout generation and personalized generation. Our code is available at https://github.com/Hatter77/AI-T2I.

Related papers

• Unified Text-Image Generation with Weakness-Targeted Post-Training [57.956648078400775]
  Unified multimodal generation architectures that jointly produce text and images have recently emerged as a promising direction for text-to-image (T2I) synthesis.<n>This work explores post-training to achieve fully unified text-image generation, where models autonomously transition from textual reasoning to visual synthesis.
  arXiv  Detail & Related papers  (2026-01-07T19:19:44Z)
• JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation [10.00677022779314]
  We propose a unified multimodal framework that encodes images and captions into discrete visual and textual tokens.<n>We incorporate cross-attention after the feature predictor for conditional denoising while maintaining a task-agnostic backbone.<n>Our approach shows that late architectural fusion combined with objective-level alignment offers an effective balance between conditioning strength and backbone generality in token-based T2I.
  arXiv  Detail & Related papers  (2025-10-01T14:51:10Z)
• Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers [79.94246924019984]
  Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress in text-driven visual generation.<n>We propose textbfTemperature-Adjusted Cross-modal Attention (TACA), a parameter-efficient method that dynamically rebalances multimodal interactions.<n>Our findings highlight the importance of balancing cross-modal attention in improving semantic fidelity in text-to-image diffusion models.
  arXiv  Detail & Related papers  (2025-06-09T17:54:04Z)
• Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects [1.419173007355272]
  Self-Cross Diffusion Guidance penalizes the overlap between cross-attention maps and the aggregated self-attention map.<n>For each subject, we aggregate self-attention maps of patches with higher cross-attention values.<n>Our method boosts the performance of both Unet-based and Transformer-based diffusion models.
  arXiv  Detail & Related papers  (2024-11-28T05:58:03Z)
• Beyond Inserting: Learning Identity Embedding for Semantic-Fidelity Personalized Diffusion Generation [21.739328335601716]
  This paper focuses on inserting accurate and interactive ID embedding into the Stable Diffusion Model for personalized generation. We propose a face-wise attention loss to fit the face region instead of entangling ID-unrelated information, such as face layout and background. Our results exhibit superior ID accuracy, text-based manipulation ability, and generalization compared to previous methods.
  arXiv  Detail & Related papers  (2024-01-31T11:52:33Z)
• Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models [58.46926334842161]
  This work illuminates the fundamental reasons for such misalignment, pinpointing issues related to low attention activation scores and mask overlaps. We propose two novel objectives, the Separate loss and the Enhance loss, that reduce object mask overlaps and maximize attention scores. Our method diverges from conventional test-time-adaptation techniques, focusing on finetuning critical parameters, which enhances scalability and generalizability.
  arXiv  Detail & Related papers  (2023-12-10T22:07:42Z)
• InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models [43.62338454684645]
  We study the problems of conditioning T2I diffusion models with Human-Object Interaction (HOI) information. We propose a pluggable interaction control model, called InteractDiffusion, that extends existing pre-trained T2I diffusion models. Our model attains the ability to control the interaction and location on existing T2I diffusion models.
  arXiv  Detail & Related papers  (2023-12-10T10:35:16Z)
• R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation [74.5598315066249]
  We probe into zero-shot grounded T2I generation with diffusion models. We propose a Region and Boundary (R&B) aware cross-attention guidance approach.
  arXiv  Detail & Related papers  (2023-10-13T05:48:42Z)
• DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability [75.9781362556431]
  We propose DiffDis to unify the cross-modal generative and discriminative pretraining into one single framework under the diffusion process. We show that DiffDis outperforms single-task models on both the image generation and the image-text discriminative tasks.
  arXiv  Detail & Related papers  (2023-08-18T05:03:48Z)
• Grounded Text-to-Image Synthesis with Attention Refocusing [16.9170825951175]
  We reveal the potential causes in the diffusion model's cross-attention and self-attention layers. We propose two novel losses to refocus attention maps according to a given spatial layout during sampling. We show that our proposed attention refocusing effectively improves the controllability of existing approaches.
  arXiv  Detail & Related papers  (2023-06-08T17:59:59Z)
• Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generators [12.053125079460234]
  We show how modern T2I generators can be used to simulate arbitrary interventions over such environmental factors. Our empirical findings demonstrate that modern T2I generators like Stable Diffusion can indeed be used as a powerful interventional data augmentation mechanism.
  arXiv  Detail & Related papers  (2022-12-21T18:07:39Z)
• Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis [78.28620571530706]
  Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. We improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions.
  arXiv  Detail & Related papers  (2022-12-09T18:30:24Z)
• What the DAAM: Interpreting Stable Diffusion Using Cross Attention [39.97805685586423]
  Large-scale diffusion neural networks represent a substantial milestone in text-to-image generation. They remain poorly understood, lacking explainability and interpretability analyses, largely due to their proprietary, closed-source nature. We propose DAAM, a novel method based on upscaling and aggregating cross-attention activations in the latent denoising subnetwork. We show that DAAM performs strongly on caption-generated images, achieving an mIoU of 61.0, and it outperforms supervised models on open-vocabulary segmentation.
  arXiv  Detail & Related papers  (2022-10-10T17:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.