MaskAttn-SDXL: Controllable Region-Level Text-To-Image Generation

• URL: http://arxiv.org/abs/2509.15357v1
• Date: Thu, 18 Sep 2025 18:57:47 GMT
• Title: MaskAttn-SDXL: Controllable Region-Level Text-To-Image Generation
• Abstract summary: We propose MaskAttn-SDXL, a region-level gating mechanism applied to the cross-attention logits of Stable Diffusion XL(SDXL)'s UNet.<n>MaskAttn-SDXL learns a binary mask per layer, injecting it into each cross-attention logit map before softmax to sparsify token-to-latent interactions.<n>Our model improves spatial compliance and attribute binding in multi-object prompts while preserving overall image quality and diversity.
• Score: 10.894281690166418
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image diffusion models achieve impressive realism but often suffer from compositional failures on prompts with multiple objects, attributes, and spatial relations, resulting in cross-token interference where entities entangle, attributes mix across objects, and spatial cues are violated. To address these failures, we propose MaskAttn-SDXL,a region-level gating mechanism applied to the cross-attention logits of Stable Diffusion XL(SDXL)'s UNet. MaskAttn-SDXL learns a binary mask per layer, injecting it into each cross-attention logit map before softmax to sparsify token-to-latent interactions so that only semantically relevant connections remain active. The method requires no positional encodings, auxiliary tokens, or external region masks, and preserves the original inference path with negligible overhead. In practice, our model improves spatial compliance and attribute binding in multi-object prompts while preserving overall image quality and diversity. These findings demonstrate that logit-level maksed cross-attention is an data-efficient primitve for enforcing compositional control, and our method thus serves as a practical extension for spatial control in text-to-image generation.

Related papers

• FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation [27.47065769615997]
  Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into pixel-level masks.<n>In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token.<n>We introduce FIRM, a Fine-grained Intra-token Representation of Masks.
  arXiv  Detail & Related papers  (2026-08-14T05:40:36Z)
• MaskAlign: Token-Subset Representation Alignment for Efficient Diffusion Training [53.777757218665386]
  MaskAlign is a token-subset representation alignment method that applies alignment to randomly sampled token subsets during training.<n>We show that MaskAlign reduces the dependence of representation alignment on the complete token set and encourages alignment behavior that is more stable under token-subset perturbations.
  arXiv  Detail & Related papers  (2026-06-07T19:11:22Z)
• SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization [64.6372217552272]
  Public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks.<n>We propose SIGMA, which performs semantic-feature differencing in a vision foundation backbone.<n>It produces a 1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets.
  arXiv  Detail & Related papers  (2026-05-27T03:55:13Z)
• YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal [83.78619790270523]
  YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module.<n>BVI selects essential tokens based on mask information enabling variable-length token processing across samples.<n>DiffSim provides a diffusion process approximation mechanism for unmasked tokens.<n>Experiments demonstrate YOSE achieves up to 2.5X speedup in 70% of cases.
  arXiv  Detail & Related papers  (2026-04-30T02:08:13Z)
• SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models [16.109077391631917]
  We show that cross-attention alone provides very coarse object localization, which however can provide initial seeds.<n>We also observe that a simple-text-guided synthetic image often has a uniform background, which is easier to find correspondences.<n>Our proposed method, dubbed SeeDiff, generates high-quality masks off-the-shelf from Stable Diffusion.
  arXiv  Detail & Related papers  (2025-07-26T05:44:00Z)
• PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions [38.32128533564591]
  Public datasets lack paired text and mask data for the same histopathology images.<n>We propose PathDiff, a diffusion framework that effectively learns from unpaired mask-text data.<n> PathDiff allows precise control over structural and contextual features, generating high-quality, semantically accurate images.
  arXiv  Detail & Related papers  (2025-06-30T00:31:03Z)
• LlamaSeg: Image Segmentation via Autoregressive Mask Generation [46.17509085054758]
  We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions.<n>We reformulate image segmentation as a visual generation problem, representing masks as "visual" tokens and employing a LLaMA-style Transformer to predict them directly from image inputs.
  arXiv  Detail & Related papers  (2025-05-26T02:22:41Z)
• SmartEraser: Remove Anything from Images using Masked-Region Guidance [114.36809682798784]
  SmartEraser is built with a new removing paradigm called Masked-Region Guidance.<n>Masked-Region Guidance retains the masked region in the input, using it as guidance for the removal process.<n>We present Syn4Removal, a large-scale object removal dataset.
  arXiv  Detail & Related papers  (2025-01-14T17:55:12Z)
• Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision [87.15580604023555]
  Unpair-Seg is a novel weakly-supervised open-vocabulary segmentation framework. It learns from unpaired image-mask and image-text pairs, which can be independently and efficiently collected. It achieves 14.6% and 19.5% mIoU on the ADE-847 and PASCAL Context-459 datasets.
  arXiv  Detail & Related papers  (2024-02-14T06:01:44Z)
• Variance-insensitive and Target-preserving Mask Refinement for Interactive Image Segmentation [68.16510297109872]
  Point-based interactive image segmentation can ease the burden of mask annotation in applications such as semantic segmentation and image editing. We introduce a novel method, Variance-Insensitive and Target-Preserving Mask Refinement to enhance segmentation quality with fewer user inputs. Experiments on GrabCut, Berkeley, SBD, and DAVIS datasets demonstrate our method's state-of-the-art performance in interactive image segmentation.
  arXiv  Detail & Related papers  (2023-12-22T02:31:31Z)
• Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation [16.863038973001483]
  This work introduces three techniques for diffusion-synthetic semantic segmentation training. First, reliability-aware robust training, originally used in weakly supervised learning, helps segmentation with insufficient synthetic mask quality. Second, large-scale pretraining of whole segmentation models, not only backbones, on synthetic ImageNet-1k-class images with pixel-labels benefits downstream segmentation tasks. Third, we introduce prompt augmentation, data augmentation to the prompt text set to scale up and diversify training images with a limited text resources.
  arXiv  Detail & Related papers  (2023-09-04T05:34:19Z)
• Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation [1.0152838128195465]
  We propose a method for spatially controlling text-to-image generation without further training of diffusion models. Our aim is to control the attention maps according to given semantic masks and text prompts.
  arXiv  Detail & Related papers  (2023-08-11T09:15:22Z)
• Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis [59.10787643285506]
  Diffusion-based models have achieved state-of-the-art performance on text-to-image synthesis tasks. One critical limitation of these models is the low fidelity of generated images with respect to the text description. We propose a new text-to-image algorithm that adds explicit control over spatial-temporal cross-attention in diffusion models.
  arXiv  Detail & Related papers  (2023-04-07T23:49:34Z)
• DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models [68.21154597227165]
  We show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model. Our approach, called DiffuMask, exploits the potential of the cross-attention map between text and image.
  arXiv  Detail & Related papers  (2023-03-21T08:43:15Z)
• Dynamic Prototype Mask for Occluded Person Re-Identification [88.7782299372656]
  Existing methods mainly address this issue by employing body clues provided by an extra network to distinguish the visible part. We propose a novel Dynamic Prototype Mask (DPM) based on two self-evident prior knowledge. Under this condition, the occluded representation could be well aligned in a selected subspace spontaneously.
  arXiv  Detail & Related papers  (2022-07-19T03:31:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.