Prompt Reinjection: Alleviating Prompt Forgetting in Multimodal Diffusion Transformers

• URL: http://arxiv.org/abs/2602.06886v1
• Date: Fri, 06 Feb 2026 17:19:53 GMT
• Title: Prompt Reinjection: Alleviating Prompt Forgetting in Multimodal Diffusion Transformers
• Authors: Yuxuan Yao, Yuxuan Chen, Hui Li, Kaihui Cheng, Qipeng Guo, Yuwei Sun, Zilong Dong, Jingdong Wang, Siyu Zhu,
• Abstract summary: Multimodal Diffusion Transformers (MMDiTs) for text-to-image generation maintain separate text and image branches.<n>We observe a prompt forgetting phenomenon: the semantics of the prompt representation in the text branch is progressively forgotten as depth increases.<n>Motivated by these findings, we introduce a training-free approach, prompt reinjection, which reinjects prompt representations from early layers into later layers to alleviate this forgetting.
• Score: 64.4017917917109
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Diffusion Transformers (MMDiTs) for text-to-image generation maintain separate text and image branches, with bidirectional information flow between text tokens and visual latents throughout denoising. In this setting, we observe a prompt forgetting phenomenon: the semantics of the prompt representation in the text branch is progressively forgotten as depth increases. We further verify this effect on three representative MMDiTs--SD3, SD3.5, and FLUX.1 by probing linguistic attributes of the representations over the layers in the text branch. Motivated by these findings, we introduce a training-free approach, prompt reinjection, which reinjects prompt representations from early layers into later layers to alleviate this forgetting. Experiments on GenEval, DPG, and T2I-CompBench++ show consistent gains in instruction-following capability, along with improvements on metrics capturing preference, aesthetics, and overall text--image generation quality.

Related papers

• Rare Text Semantics Were Always There in Your Diffusion Transformer [14.05664612353265]
  We propose a simple yet effective intervention that surfaces rare semantics inside Multi-modal Diffusion Transformers (MM-DiTs)<n>In particular, the joint-attention mechanism intrinsic to MM-DiT sequentially updates text embeddings alongside image embeddings throughout transformer blocks.<n>Our results generalize effectively across text-to-vision tasks, including text-to-image, text-to-video, and text-driven image editing.
  arXiv  Detail & Related papers  (2025-10-04T17:41:24Z)
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers [56.76198904599581]
  Text-to-image diffusion models excel at translating language prompts into implicitly grounding concepts through their cross-modal attention mechanisms.<n>Recent multi-modal diffusion transformers extend this by introducing joint self-attentiond image and text tokens, enabling richer and more scalable cross-modal alignment.<n>We introduce Seg4Diff, a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image.
  arXiv  Detail & Related papers  (2025-09-22T17:59:54Z)
• VSC: Visual Search Compositional Text-to-Image Diffusion Model [15.682990658945682]
  We introduce a novel compositional generation method that leverages pairwise image embeddings to improve attribute-object binding.<n>Our approach decomposes complex prompts into sub-prompts, generates corresponding images, and computes visual prototypes that fuse with text embeddings to enhance representation.<n>Our approaches outperform existing compositional text-to-image diffusion models on the benchmark T2I CompBench, achieving better image quality, evaluated by humans, and emerging robustness under scaling number of binding pairs in the prompt.
  arXiv  Detail & Related papers  (2025-05-02T08:31:43Z)
• Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines [33.49257838597258]
  Text-to-image diffusion models (T2I) use a latent representation of a text prompt to guide the image generation process. We propose the Diffusion Lens, a method for analyzing the text encoder of T2I models by generating images from its intermediate representations.
  arXiv  Detail & Related papers  (2024-03-09T09:11:49Z)
• Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models [68.47333676663312]
  We show a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models. The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens. We illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.
  arXiv  Detail & Related papers  (2024-02-21T03:01:17Z)
• Learning from Mistakes: Iterative Prompt Relabeling for Text-to-Image Diffusion Model Training [33.51524424536508]
  Iterative Prompt Relabeling (IPR) is a novel algorithm that aligns images to text through iterative image sampling and prompt relabeling with feedback.<n>We conduct thorough experiments on SDv2 and SDXL, testing their capability to follow instructions on spatial relations.
  arXiv  Detail & Related papers  (2023-12-23T11:10:43Z)
• Image Super-Resolution with Text Prompt Diffusion [118.023531454099]
  We introduce text prompts to image SR to provide degradation priors.<n>PromptSR leverages the latest multi-modal large language model (MLLM) to generate prompts from low-resolution images.<n>Experiments indicate that introducing text prompts into SR, yields impressive results on both synthetic and real-world images.
  arXiv  Detail & Related papers  (2023-11-24T05:11:35Z)
• PV2TEA: Patching Visual Modality to Textual-Established Information Extraction [59.76117533540496]
  We patch the visual modality to the textual-established attribute information extractor. PV2TEA is an encoder-decoder architecture equipped with three bias reduction schemes. Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
  arXiv  Detail & Related papers  (2023-06-01T05:39:45Z)
• Enhanced Modality Transition for Image Captioning [51.72997126838352]
  We build a Modality Transition Module (MTM) to transfer visual features into semantic representations before forwarding them to the language model. During the training phase, the modality transition network is optimised by the proposed modality loss. Experiments have been conducted on the MS-COCO dataset demonstrating the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2021-02-23T07:20:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.