FocusDiT: Masking Queries in Diffusion Transformers for Fine-grained Image Generation

• URL: http://arxiv.org/abs/2606.02090v2
• Date: Tue, 02 Jun 2026 15:33:35 GMT
• Title: FocusDiT: Masking Queries in Diffusion Transformers for Fine-grained Image Generation
• Abstract summary: Diffusion transformer (DiT) has been widely adopted in the generative diffusion field.<n>We present that focusing on critical query tokens corresponding to more complex details is essential for fine-grained visual generation.<n>We propose FocusDiT, which applies a Masking scheme to focus on critical query tokens that are exclusively fed into FFN.
• Score: 64.71746346442445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion transformer (DiT) has been widely adopted in the generative diffusion field, advancing the denoising of query tokens through attention and Feed-Forward (\text{FFN}) layers. FFN actually acts as the key-value vocabulary for decoding visual contents where the value embeds the visual semantical knowledge. We present that focusing on critical query tokens corresponding to more complex details and encouraging the model to improve these tokens is essential for fine-grained visual generation. To this end, we propose FocusDiT, which applies a Masking scheme to focus on critical query tokens that are exclusively fed into FFN. The masked queries can retrieve visual tokens from the FFN vocabularies, and use them to decode their visual details. Extensive text-to-image experiments validate the effectiveness of token masking in enhancing generative performance.

Related papers

• Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers [16.996490884328455]
  Text-to-image diffusion transformers (DiTs) generate images through joint attention.<n>In large-scale DiTs, the conditioning input contains not only the user prompt but also chat-template tokens.<n>We introduce a causal interpretability framework to probe how these tokens participate in the denoising computation.
  arXiv  Detail & Related papers  (2026-07-21T14:29:59Z)
• Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models [40.902681492117786]
  RemeDi is a mask-based DLM that predicts token distributions and per-token confidence scores at each step.<n>We train a remask-aware pipeline to train this ability, including supervised fine-tuning which teaches the model to detect and remask incorrect tokens.<n>Experiments show that RemeDi achieves the state-of-the-art results among open-source DLMs on multiple datasets.
  arXiv  Detail & Related papers  (2025-09-28T05:39:49Z)
• Rethinking Visual Token Reduction in LVLMs under Cross-modal Misalignment [38.04426918886084]
  Vision-Language Models (LVLMs) encode visual inputs as dense sequences of patch-level tokens to capture fine-grained semantics.<n>Previous efforts have explored visual token reduction either prior to or within the large language models (LLMs)<n>We introduce VisionDrop, a training-free, visual-only pruning framework that selects informative visual tokens based on intra-modal (visual-to-visual) attention.
  arXiv  Detail & Related papers  (2025-06-27T14:55:40Z)
• Efficient Multi-modal Large Language Models via Visual Token Grouping [55.482198808206284]
  High-resolution images and videos pose a barrier to their broader adoption.<n> compressing vision tokens in MLLMs has emerged as a promising approach to reduce inference costs.<n>We introduce VisToG, a novel grouping mechanism that leverages the capabilities of pre-trained vision encoders to group similar image segments.
  arXiv  Detail & Related papers  (2024-11-26T09:36:02Z)
• KNN Transformer with Pyramid Prompts for Few-Shot Learning [52.735070934075736]
  Few-Shot Learning aims to recognize new classes with limited labeled data. Recent studies have attempted to address the challenge of rare samples with textual prompts to modulate visual features.
  arXiv  Detail & Related papers  (2024-10-14T07:39:30Z)
• Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning [40.08368469646114]
  We provide semantically-meaningful visual tokens to transformer encoders within a vision-language pre-training framework.<n>We demonstrate notable improvements over ViTs in learned representation quality across text-to-image and image-to-text retrieval tasks.
  arXiv  Detail & Related papers  (2024-05-26T01:46:22Z)
• Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models [1.6450779686641077]
  We introduce Open-Vocabulary Attention Maps (OVAM)-a training-free method for text-to-image diffusion models. We evaluate these tokens within existing state-of-the-art Stable Diffusion extensions.
  arXiv  Detail & Related papers  (2024-03-21T10:56:12Z)
• MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask [84.84034179136458]
  A crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning. We propose an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models.
  arXiv  Detail & Related papers  (2023-09-08T15:53:37Z)
• Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling [61.03262873980619]
  Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. We propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model.
  arXiv  Detail & Related papers  (2021-11-24T18:50:47Z)
• Rethinking Global Context in Crowd Counting [70.54184500538338]
  A pure transformer is used to extract features with global information from overlapping image patches. Inspired by classification, we add a context token to the input sequence, to facilitate information exchange with tokens corresponding to image patches.
  arXiv  Detail & Related papers  (2021-05-23T12:44:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.