Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis

• URL: http://arxiv.org/abs/2503.22168v1
• Date: Fri, 28 Mar 2025 06:12:25 GMT
• Title: Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis
• Authors: Woojung Han, Yeonkyung Lee, Chanyoung Kim, Kwanghyun Park, Seong Jae Hwang,
• Abstract summary: Diffusion-based text-to-image (T2I) models have excelled in high-quality image generation.<n>We propose STORM, a novel training-free approach for spatially coherent T2I synthesis.
• Score: 5.869767284889891
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Diffusion-based text-to-image (T2I) models have recently excelled in high-quality image generation, particularly in a training-free manner, enabling cost-effective adaptability and generalization across diverse tasks. However, while the existing methods have been continuously focusing on several challenges, such as "missing objects" and "mismatched attributes," another critical issue of "mislocated objects" remains where generated spatial positions fail to align with text prompts. Surprisingly, ensuring such seemingly basic functionality remains challenging in popular T2I models due to the inherent difficulty of imposing explicit spatial guidance via text forms. To address this, we propose STORM (Spatial Transport Optimization by Repositioning Attention Map), a novel training-free approach for spatially coherent T2I synthesis. STORM employs Spatial Transport Optimization (STO), rooted in optimal transport theory, to dynamically adjust object attention maps for precise spatial adherence, supported by a Spatial Transport (ST) Cost function that enhances spatial understanding. Our analysis shows that integrating spatial awareness is most effective in the early denoising stages, while later phases refine details. Extensive experiments demonstrate that STORM surpasses existing methods, effectively mitigating mislocated objects while improving missing and mismatched attributes, setting a new benchmark for spatial alignment in T2I synthesis.

Related papers

• ESPLoRA: Enhanced Spatial Precision with Low-Rank Adaption in Text-to-Image Diffusion Models for High-Definition Synthesis [45.625062335269355]
  Diffusion models have revolutionized text-to-image (T2I) synthesis, producing high-quality, photorealistic images. However, they still struggle to properly render the spatial relationships described in text prompts. Our approach builds upon a curated dataset of spatially explicit prompts, meticulously extracted and synthesized from LAION-400M. We present ESPLoRA, a flexible fine-tuning framework based on Low-Rank Adaptation, to enhance spatial consistency in generative models.
  arXiv  Detail & Related papers  (2025-04-18T15:21:37Z)
• Optimal Transport Adapter Tuning for Bridging Modality Gaps in Few-Shot Remote Sensing Scene Classification [80.83325513157637]
  Few-Shot Remote Sensing Scene Classification (FS-RSSC) presents the challenge of classifying remote sensing images with limited labeled samples. We propose a novel Optimal Transport Adapter Tuning (OTAT) framework aimed at constructing an ideal Platonic representational space.
  arXiv  Detail & Related papers  (2025-03-19T07:04:24Z)
• CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models [13.992486106252716]
  CoMPaSS is a versatile training framework that enhances spatial understanding of any T2I diffusion model.<n>CoMPaSS solves the ambiguity of spatial-related data with the Spatial Constraints-Oriented Pairing (SCOP) data engine.<n>To better exploit the curated high-quality spatial priors, CoMPaSS introduces a Token ENcoding ORdering (TENOR) module.
  arXiv  Detail & Related papers  (2024-12-17T18:59:50Z)
• HSLiNets: Hyperspectral Image and LiDAR Data Fusion Using Efficient Dual Non-Linear Feature Learning Networks [7.06787067270941]
  The integration of hyperspectral imaging (HSI) and LiDAR data within new linear feature spaces offers a promising solution to the challenges posed by the high-dimensionality and redundancy inherent in HSIs. This study introduces a dual linear fused space framework that capitalizes on bidirectional reversed convolutional neural network (CNN) pathways, coupled with a specialized spatial analysis block. The proposed method not only enhances data processing and classification accuracy, but also mitigates the computational burden typically associated with advanced models such as Transformers.
  arXiv  Detail & Related papers  (2024-11-30T01:08:08Z)
• StarVid: Enhancing Semantic Alignment in Video Diffusion Models via Spatial and SynTactic Guided Attention Refocusing [40.50917266880829]
  We propose textbfStarVid, a plug-and-play, training-free method that improves semantic alignment between multiple subjects, their motions, and text prompts in T2V models.<n>StarVid first leverages the spatial reasoning capabilities of large language models (LLMs) for two-stage motion trajectory planning based on text prompts.
  arXiv  Detail & Related papers  (2024-09-23T17:56:03Z)
• Getting it Right: Improving Spatial Consistency in Text-to-Image Models [103.52640413616436]
  One of the key shortcomings in current text-to-image (T2I) models is their inability to consistently generate images which faithfully follow the spatial relationships specified in the text prompt. We create SPRIGHT, the first spatially focused, large-scale dataset, by re-captioning 6 million images from 4 widely used vision datasets. We find that training on images containing a larger number of objects leads to substantial improvements in spatial consistency, including state-of-the-art results on T2I-CompBench with a spatial score of 0.2133, by fine-tuning on 500 images.
  arXiv  Detail & Related papers  (2024-04-01T15:55:25Z)
• SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation [68.42476385214785]
  We propose a novel Spatial-Semantic Map Guided (SSMG) diffusion model that adopts the feature map, derived from the layout, as guidance. SSMG achieves superior generation quality with sufficient spatial and semantic controllability compared to previous works. We also propose the Relation-Sensitive Attention (RSA) and Location-Sensitive Attention (LSA) mechanisms.
  arXiv  Detail & Related papers  (2023-08-20T04:09:12Z)
• Spatial-Aware Token for Weakly Supervised Object Localization [137.0570026552845]
  We propose a task-specific spatial-aware token to condition localization in a weakly supervised manner. Experiments show that the proposed SAT achieves state-of-the-art performance on both CUB-200 and ImageNet, with 98.45% and 73.13% GT-known Loc.
  arXiv  Detail & Related papers  (2023-03-18T15:38:17Z)
• Benchmarking Spatial Relationships in Text-to-Image Generation [102.62422723894232]
  We investigate the ability of text-to-image models to generate correct spatial relationships among objects. We present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image. Our experiments reveal a surprising finding that, although state-of-the-art T2I models exhibit high image quality, they are severely limited in their ability to generate multiple objects or the specified spatial relations between them.
  arXiv  Detail & Related papers  (2022-12-20T06:03:51Z)
• Maximum Spatial Perturbation Consistency for Unpaired Image-to-Image Translation [56.44946660061753]
  This paper proposes a universal regularization technique called maximum spatial perturbation consistency (MSPC) MSPC enforces a spatial perturbation function (T ) and the translation operator (G) to be commutative (i.e., TG = GT ) Our method outperforms the state-of-the-art methods on most I2I benchmarks.
  arXiv  Detail & Related papers  (2022-03-23T19:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.