Unraveling MMDiT Blocks: Training-free Analysis and Enhancement of Text-conditioned Diffusion

• URL: http://arxiv.org/abs/2601.02211v1
• Date: Mon, 05 Jan 2026 15:32:53 GMT
• Title: Unraveling MMDiT Blocks: Training-free Analysis and Enhancement of Text-conditioned Diffusion
• Abstract summary: We study how different blocks and their interactions with textual conditions contribute to the synthesis process.<n>Our analysis reveals that semantic information appears in earlier blocks and finer details are rendered in later blocks.<n>We propose novel training-free strategies for improved text alignment, precise editing, and acceleration.
• Score: 41.431041419460584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent breakthroughs of transformer-based diffusion models, particularly with Multimodal Diffusion Transformers (MMDiT) driven models like FLUX and Qwen Image, have facilitated thrilling experiences in text-to-image generation and editing. To understand the internal mechanism of MMDiT-based models, existing methods tried to analyze the effect of specific components like positional encoding and attention layers. Yet, a comprehensive understanding of how different blocks and their interactions with textual conditions contribute to the synthesis process remains elusive. In this paper, we first develop a systematic pipeline to comprehensively investigate each block's functionality by removing, disabling and enhancing textual hidden-states at corresponding blocks. Our analysis reveals that 1) semantic information appears in earlier blocks and finer details are rendered in later blocks, 2) removing specific blocks is usually less disruptive than disabling text conditions, and 3) enhancing textual conditions in selective blocks improves semantic attributes. Building on these observations, we further propose novel training-free strategies for improved text alignment, precise editing, and acceleration. Extensive experiments demonstrated that our method outperforms various baselines and remains flexible across text-to-image generation, image editing, and inference acceleration. Our method improves T2I-Combench++ from 56.92% to 63.00% and GenEval from 66.42% to 71.63% on SD3.5, without sacrificing synthesis quality. These results advance understanding of MMDiT models and provide valuable insights to unlock new possibilities for further improvements.

Related papers

• Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement [13.410085175938486]
  Medical image synthesis is crucial for alleviating data scarcity and privacy constraints.<n>Fine-tuning general text-to-image (T2I) models remains challenging.<n>We propose a Visually-Guided Text Disentanglement framework.
  arXiv  Detail & Related papers  (2026-03-11T08:20:33Z)
• Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation [41.431041419460584]
  Diff-Aid is a lightweight inference-time method that adjusts per-token text and image interactions across transformer blocks and denoising timesteps.<n>As a plug-and-play module, Diff-Aid can be seamlessly integrated into downstream applications for further improvement.
  arXiv  Detail & Related papers  (2026-02-14T04:08:05Z)
• Yuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blocks [28.895073514108088]
  Diffusion models have shown remarkable capacity in image synthesis based on their U-shaped architecture and convolutional neural networks (CNN) as basic blocks.<n>We propose a text-conditioned diffusion model with Swin-transformer in this work.<n>Yuan-TecSwin achieves the state-of-the-art FID score of 1.37 on ImageNet generation benchmark, without any additional models at different denoising stages.
  arXiv  Detail & Related papers  (2025-12-18T14:32:06Z)
• Feedback Descent: Open-Ended Text Optimization via Pairwise Comparison [48.89195616081196]
  Feedback Descent is a framework that optimize text artifacts -- prompts, code, and molecules -- through structured textual feedback.<n>We show that in-context learning can transform structured feedback into gradient-like directional information, enabling targeted edits.<n>In the DOCKSTRING molecule discovery benchmark, Feedback Descent identifies novel drug-like molecules surpassing the $99.9$th percentile of a database with more than $260,000$ compounds across six protein targets.
  arXiv  Detail & Related papers  (2025-11-11T07:14:13Z)
• Rare Text Semantics Were Always There in Your Diffusion Transformer [14.05664612353265]
  We propose a simple yet effective intervention that surfaces rare semantics inside Multi-modal Diffusion Transformers (MM-DiTs)<n>In particular, the joint-attention mechanism intrinsic to MM-DiT sequentially updates text embeddings alongside image embeddings throughout transformer blocks.<n>Our results generalize effectively across text-to-vision tasks, including text-to-image, text-to-video, and text-driven image editing.
  arXiv  Detail & Related papers  (2025-10-04T17:41:24Z)
• STRICT: Stress Test of Rendering Images Containing Text [14.124910427202273]
  $textbfSTRICT$ is a benchmark designed to stress-test the ability of diffusion models to render coherent and instruction-aligned text in images.<n>We evaluate several state-of-the-art models, including proprietary and open-source variants, and reveal persistent limitations in long-range consistency and instruction-following capabilities.
  arXiv  Detail & Related papers  (2025-05-25T05:37:08Z)
• Mimir: Improving Video Diffusion Models for Precise Text Understanding [53.72393225042688]
  Text serves as the key control signal in video generation due to its narrative nature.<n>The recent success of large language models (LLMs) showcases the power of decoder-only transformers.<n>This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser.
  arXiv  Detail & Related papers  (2024-12-04T07:26:44Z)
• Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation [30.99444221104659]
  We find that the latest Multimodal Diffusion Transformer (MMDiT) still suffers from subject ambiguity or mixing when the input text prompt contains multiple subjects of similar semantics or appearance.<n>We identify three possible ambiguities that cause this problem: Inter-block Ambiguity, Text Ambiguity, and Semantic Ambiguity.<n>We propose to repair the ambiguous latent on-the-fly by test-time optimization at early denoising steps.
  arXiv  Detail & Related papers  (2024-11-27T12:47:06Z)
• ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models [52.23899502520261]
  We introduce a novel framework named, ARTIST, which incorporates a dedicated textual diffusion model to focus on the learning of text structures specifically.<n>We finetune a visual diffusion model, enabling it to assimilate textual structure information from the pretrained textual model.<n>This disentangled architecture design and training strategy significantly enhance the text rendering ability of the diffusion models for text-rich image generation.
  arXiv  Detail & Related papers  (2024-06-17T19:31:24Z)
• Enhancing Text-to-Image Editing via Hybrid Mask-Informed Fusion [61.42732844499658]
  This paper systematically improves the text-guided image editing techniques based on diffusion models. We incorporate human annotation as an external knowledge to confine editing within a Mask-informed'' region.
  arXiv  Detail & Related papers  (2024-05-24T07:53:59Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.