Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models

• URL: http://arxiv.org/abs/2510.23974v1
• Date: Tue, 28 Oct 2025 01:10:15 GMT
• Title: Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models
• Authors: Byeonghu Na, Minsang Park, Gyuwon Sim, Donghyeok Shin, HeeSun Bae, Mina Kang, Se Jung Kwon, Wanmo Kang, Il-Chul Moon,
• Abstract summary: We propose Diffusion Adaptive Text Embedding (DATE), which dynamically updates text embeddings at each diffusion timestep based on intermediate perturbed data.<n>We show that DATE maintains the generative capability of the model while providing superior text-image alignment over fixed text embeddings across various tasks.
• Score: 33.043266237235606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-image diffusion models rely on text embeddings from a pre-trained text encoder, but these embeddings remain fixed across all diffusion timesteps, limiting their adaptability to the generative process. We propose Diffusion Adaptive Text Embedding (DATE), which dynamically updates text embeddings at each diffusion timestep based on intermediate perturbed data. We formulate an optimization problem and derive an update rule that refines the text embeddings at each sampling step to improve alignment and preference between the mean predicted image and the text. This allows DATE to dynamically adapts the text conditions to the reverse-diffused images throughout diffusion sampling without requiring additional model training. Through theoretical analysis and empirical results, we show that DATE maintains the generative capability of the model while providing superior text-image alignment over fixed text embeddings across various tasks, including multi-concept generation and text-guided image editing. Our code is available at https://github.com/aailab-kaist/DATE.

Related papers

• Instructing Text-to-Image Diffusion Models via Classifier-Guided Semantic Optimization [48.38187112651368]
  We propose optimizing semantic embeddings guided by attribute classifiers to steer text-to-image models toward desired edits.<n>Our method achieves high levels of disentanglement and strong generalization across different domains of data.
  arXiv  Detail & Related papers  (2025-05-20T12:07:01Z)
• Textualize Visual Prompt for Image Editing via Diffusion Bridge [15.696208035498753]
  Current visual prompt methods rely on a pretrained text-guided image-to-image generative model.<n>We present a framework based on any single text-to-image model without reliance on the explicit image-to-image model.
  arXiv  Detail & Related papers  (2025-01-07T03:33:22Z)
• Contextualized Diffusion Models for Text-Guided Image and Video Generation [67.69171154637172]
  Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. We propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample. We generalize our model to both DDPMs and DDIMs with theoretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing.
  arXiv  Detail & Related papers  (2024-02-26T15:01:16Z)
• Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering [118.53208190209517]
  We propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. Our method can effectively learn the prompts to improve the matches between the input text and the generated images.
  arXiv  Detail & Related papers  (2024-01-12T03:46:29Z)
• UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models [25.219960711604728]
  This paper proposes a novel approach for text image generation, utilizing a pre-trained diffusion model. Our approach involves the design and training of a light-weight character-level text encoder, which replaces the original CLIP encoder. By employing an inference stage refinement process, we achieve a notably high sequence accuracy when synthesizing text in arbitrarily given images.
  arXiv  Detail & Related papers  (2023-12-08T07:47:46Z)
• Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models [63.99110667987318]
  We present DiffText, a pipeline that seamlessly blends foreground text with the background's intrinsic features. With fewer text instances, our produced text images consistently surpass other synthetic data in aiding text detectors.
  arXiv  Detail & Related papers  (2023-11-28T06:51:28Z)
• Enhancing Diffusion Models with Text-Encoder Reinforcement Learning [63.41513909279474]
  Text-to-image diffusion models are typically trained to optimize the log-likelihood objective. Recent research addresses this issue by refining the diffusion U-Net using human rewards through reinforcement learning or direct backpropagation. We demonstrate that by finetuning the text encoder through reinforcement learning, we can enhance the text-image alignment of the results.
  arXiv  Detail & Related papers  (2023-11-27T09:39:45Z)
• PRedItOR: Text Guided Image Editing with Diffusion Prior [2.3022070933226217]
  Text guided image editing requires compute intensive optimization of text embeddings or fine-tuning the model weights for text guided image editing. Our architecture consists of a diffusion prior model that generates CLIP image embedding conditioned on a text prompt and a custom Latent Diffusion Model trained to generate images conditioned on CLIP image embedding. We combine this with structure preserving edits on the image decoder using existing approaches such as reverse DDIM to perform text guided image editing.
  arXiv  Detail & Related papers  (2023-02-15T22:58:11Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.