Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

• URL: http://arxiv.org/abs/2607.19139v2
• Date: Thu, 30 Jul 2026 16:07:04 GMT
• Title: Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
• Abstract summary: Text-to-image diffusion transformers (DiTs) generate images through joint attention.<n>In large-scale DiTs, the conditioning input contains not only the user prompt but also chat-template tokens.<n>We introduce a causal interpretability framework to probe how these tokens participate in the denoising computation.
• Score: 16.996490884328455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern text-to-image diffusion transformers (DiTs) generate images through joint attention, in which text and image tokens interact directly within a single sequence. In large-scale DiTs, the conditioning input contains not only the user prompt but also chat-template tokens introduced by LLM-based text encoders. Yet how these tokens participate in the denoising computation remains poorly understood. To probe this, we introduce a causal interpretability framework. Using it to separate prompt-content tokens from chat-template tokens, we find that the template tokens carry little prompt-specific information at the encoder output. Yet surprisingly, they emerge as dominant image-to-text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers. We show that they acquire this identity indirectly. Rather than reading the prompt tokens, they draw the identity from the image latents into which the prompt semantics have already been injected at the very first layer. We further reveal a division of labor across heads and depth in DiTs, where distinct heads route semantics or render visual structure, and identity is committed in early blocks, carried by middle blocks, and refined in late ones. As a practical payoff, this analysis yields a training-free pruning rule that removes the causally inert prompt-reading heads and cuts $20\%$ of joint-attention FLOPs at a $1.4$-point cost in GenEval accuracy. Overall, our work not only reveals that the tokens encoding semantics at the input need not be those that maintain them during generation, but also provides a causal view of internal mechanisms in diffusion transformers.

Related papers

• Token-to-Token Alignment of Text Embeddings for Semantic Blending [55.88554156801394]
  We introduce Token-to-Token alignment, a framework that establishes explicit semantic correspondence between tokens across prompts.<n>Our results show that text embedding spaces in text-to-image models implicitly encode a continuous semantic structure that becomes accessible once representations are properly aligned.
  arXiv  Detail & Related papers  (2026-06-22T23:54:40Z)
• FocusDiT: Masking Queries in Diffusion Transformers for Fine-grained Image Generation [64.71746346442445]
  Diffusion transformer (DiT) has been widely adopted in the generative diffusion field.<n>We present that focusing on critical query tokens corresponding to more complex details is essential for fine-grained visual generation.<n>We propose FocusDiT, which applies a Masking scheme to focus on critical query tokens that are exclusively fed into FFN.
  arXiv  Detail & Related papers  (2026-06-01T11:18:11Z)
• Vision-Language Binding in In-Context Image Generation [34.419605761529404]
  In-context image generation models such as FLUX.2 take a text prompt and an optional reference image as visual conditioning for the output.<n>We show that an implicit cross-modal binding emerges between the text tokens and the reference image.<n>We further localize the reference-text binding to the padding tokens of the text sequence.
  arXiv  Detail & Related papers  (2026-05-23T15:30:19Z)
• Taming Outlier Tokens in Diffusion Transformers [55.42341508886889]
  We study outlier tokens in Diffusion Transformers (DiTs) for image generation.<n>We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines.<n>We introduce Dual-Stage Registers (DSR), a register-based intervention for both components.
  arXiv  Detail & Related papers  (2026-05-06T17:59:42Z)
• Prompt Reinjection: Alleviating Prompt Forgetting in Multimodal Diffusion Transformers [64.4017917917109]
  Multimodal Diffusion Transformers (MMDiTs) for text-to-image generation maintain separate text and image branches.<n>We observe a prompt forgetting phenomenon: the semantics of the prompt representation in the text branch is progressively forgotten as depth increases.<n>Motivated by these findings, we introduce a training-free approach, prompt reinjection, which reinjects prompt representations from early layers into later layers to alleviate this forgetting.
  arXiv  Detail & Related papers  (2026-02-06T17:19:53Z)
• Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models [35.85433370296494]
  We investigate how semantic information is distributed across token representations in a text-to-image model.<n>We find information is usually concentrated in only one or two of the item's tokens.<n>In some cases, items do influence each other's representation, often leading to misinterpretations.
  arXiv  Detail & Related papers  (2025-04-01T19:17:44Z)
• Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models [64.52046218688295]
  Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process.<n>We conduct the first in-depth analysis of the role padding tokens play in T2I models.<n>Our findings reveal three distinct scenarios: padding tokens may affect the model's output during text encoding, during the diffusion process, or be effectively ignored.
  arXiv  Detail & Related papers  (2025-01-12T08:36:38Z)
• Latent Space Disentanglement in Diffusion Transformers Enables Precise Zero-shot Semantic Editing [4.948910649137149]
  Diffusion Transformers (DiTs) have recently achieved remarkable success in text-guided image generation. We show how multimodal information collectively forms this joint space and how they guide the semantics of the synthesized images. We propose a simple yet effective Encode-Identify-Manipulate (EIM) framework for zero-shot fine-grained image editing.
  arXiv  Detail & Related papers  (2024-11-12T21:34:30Z)
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens [49.569695524535454]
  We propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis.
  arXiv  Detail & Related papers  (2024-07-07T15:16:19Z)
• Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning [40.08368469646114]
  We provide semantically-meaningful visual tokens to transformer encoders within a vision-language pre-training framework.<n>We demonstrate notable improvements over ViTs in learned representation quality across text-to-image and image-to-text retrieval tasks.
  arXiv  Detail & Related papers  (2024-05-26T01:46:22Z)
• MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask [84.84034179136458]
  A crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning. We propose an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models.
  arXiv  Detail & Related papers  (2023-09-08T15:53:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.