Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation

• URL: http://arxiv.org/abs/2602.13585v1
• Date: Sat, 14 Feb 2026 04:08:05 GMT
• Title: Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation
• Abstract summary: Diff-Aid is a lightweight inference-time method that adjusts per-token text and image interactions across transformer blocks and denoising timesteps.<n>As a plug-and-play module, Diff-Aid can be seamlessly integrated into downstream applications for further improvement.
• Score: 41.431041419460584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions between textual and visual features. Prior approaches enhance such interactions via architectural design or handcrafted textual condition weighting, but lack flexibility and overlook the dynamic interactions across different blocks and denoising stages. To provide a more flexible and efficient solution to this problem, we propose Diff-Aid, a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps. Beyond improving generation quality, Diff-Aid yields interpretable modulation patterns that reveal how different blocks, timesteps, and textual tokens contribute to semantic alignment during denoising. As a plug-and-play module, Diff-Aid can be seamlessly integrated into downstream applications for further improvement, including style LoRAs, controllable generation, and zero-shot editing. Experiments on strong baselines (SD 3.5 and FLUX) demonstrate consistent improvements in prompt adherence, visual quality, and human preference across various metrics. Our code and models will be released.

Related papers

• TextLDM: Language Modeling with Continuous Latent Diffusion [89.69255520673248]
  Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos.<n>We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification.
  arXiv  Detail & Related papers  (2026-05-08T13:54:34Z)
• Unraveling MMDiT Blocks: Training-free Analysis and Enhancement of Text-conditioned Diffusion [41.431041419460584]
  We study how different blocks and their interactions with textual conditions contribute to the synthesis process.<n>Our analysis reveals that semantic information appears in earlier blocks and finer details are rendered in later blocks.<n>We propose novel training-free strategies for improved text alignment, precise editing, and acceleration.
  arXiv  Detail & Related papers  (2026-01-05T15:32:53Z)
• GAID: Frame-Level Gated Audio-Visual Integration with Directional Perturbation for Text-Video Retrieval [12.483734449829235]
  GAID is a framework that integrates audio and visual features under textual guidance.<n>DASP injects structure-aware perturbations into text embeddings, enhancing robustness and discrimination without incurring multi-pass inference.<n>Experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX show consistent state-of-the-art results with notable efficiency gains.
  arXiv  Detail & Related papers  (2025-08-03T10:44:24Z)
• Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers [79.94246924019984]
  Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress in text-driven visual generation.<n>We propose textbfTemperature-Adjusted Cross-modal Attention (TACA), a parameter-efficient method that dynamically rebalances multimodal interactions.<n>Our findings highlight the importance of balancing cross-modal attention in improving semantic fidelity in text-to-image diffusion models.
  arXiv  Detail & Related papers  (2025-06-09T17:54:04Z)
• SALAD: Skeleton-aware Latent Diffusion for Text-driven Motion Generation and Editing [5.123822132804602]
  We introduce a skeleton-aware latent diffusion (SALAD) model that captures the intricate inter-relationships between joints, frames, and words.<n>By leveraging cross-attention maps produced during the generation process, we enable attention-based zero-shot text-driven motion editing.<n>Our approach significantly outperforms previous methods in terms of text-motion alignment without compromising generation quality.
  arXiv  Detail & Related papers  (2025-03-18T02:20:11Z)
• Contextualized Diffusion Models for Text-Guided Image and Video Generation [67.69171154637172]
  Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. We propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample. We generalize our model to both DDPMs and DDIMs with theoretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing.
  arXiv  Detail & Related papers  (2024-02-26T15:01:16Z)
• SemanticBoost: Elevating Motion Generation with Augmented Textual Cues [73.83255805408126]
  Our framework comprises a Semantic Enhancement module and a Context-Attuned Motion Denoiser (CAMD) The CAMD approach provides an all-encompassing solution for generating high-quality, semantically consistent motion sequences. Our experimental results demonstrate that SemanticBoost, as a diffusion-based method, outperforms auto-regressive-based techniques.
  arXiv  Detail & Related papers  (2023-10-31T09:58:11Z)
• MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask [84.84034179136458]
  A crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning. We propose an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models.
  arXiv  Detail & Related papers  (2023-09-08T15:53:37Z)
• Learning to Model Multimodal Semantic Alignment for Story Visualization [58.16484259508973]
  Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. We explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
  arXiv  Detail & Related papers  (2022-11-14T11:41:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.