GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models

• URL: http://arxiv.org/abs/2512.15560v1
• Date: Wed, 17 Dec 2025 16:09:43 GMT
• Title: GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models
• Authors: Bozhou Li, Sihan Yang, Yushuo Guan, Ruichuan An, Xinlong Chen, Yang Shi, Pengfei Wan, Wentao Zhang, Yuanxing zhang,
• Abstract summary: We introduce GRAN-TED, a paradigm to Generate Robust, Aligned, andAlignedd Text Embeddings for Diffusion models.<n>We propose TED-6K, a novel text-only benchmark that enables efficient and robust assessment of an encoder's representational quality.<n>We develop a superior text encoder using a novel two-stage training paradigm.
• Score: 20.650166688664115
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The text encoder is a critical component of text-to-image and text-to-video diffusion models, fundamentally determining the semantic fidelity of the generated content. However, its development has been hindered by two major challenges: the lack of an efficient evaluation framework that reliably predicts downstream generation performance, and the difficulty of effectively adapting pretrained language models for visual synthesis. To address these issues, we introduce GRAN-TED, a paradigm to Generate Robust, Aligned, and Nuanced Text Embeddings for Diffusion models. Our contribution is twofold. First, we propose TED-6K, a novel text-only benchmark that enables efficient and robust assessment of an encoder's representational quality without requiring costly end-to-end model training. We demonstrate that performance on TED-6K, standardized via a lightweight, unified adapter, strongly correlates with an encoder's effectiveness in downstream generation tasks. Second, guided by this validated framework, we develop a superior text encoder using a novel two-stage training paradigm. This process involves an initial fine-tuning stage on a Multimodal Large Language Model for better visual representation, followed by a layer-wise weighting method to extract more nuanced and potent text features. Our experiments show that the resulting GRAN-TED encoder not only achieves state-of-the-art performance on TED-6K but also leads to demonstrable performance gains in text-to-image and text-to-video generation. Our code is available at the following link: https://anonymous.4open.science/r/GRAN-TED-4FCC/.

Related papers

• CoF-T2I: Video Models as Pure Visual Reasoners for Text-to-Image Generation [52.0601996237501]
  Chain-of-Frame (CoF) reasoning enables frame-by-frame visual inference.<n>CoF-T2I integrates CoF reasoning into text-to-image (T2I) generation via progressive visual refinement.<n>Experiments show that CoF-T2I significantly outperforms the base video model.
  arXiv  Detail & Related papers  (2026-01-15T04:33:06Z)
• Decoder-Only LLMs are Better Controllers for Diffusion Models [63.22040456010123]
  We propose to enhance text-to-image diffusion models by borrowing the strength of semantic understanding from large language models.<n>Our adapter module is superior to the stat-of-the-art models in terms of text-to-image generation quality and reliability.
  arXiv  Detail & Related papers  (2025-02-06T12:17:35Z)
• TextCraftor: Your Text Encoder Can be Image Quality Controller [65.27457900325462]
  Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have revolutionized the field of content generation. We propose a proposed fine-tuning approach, TextCraftor, to enhance the performance of text-to-image diffusion models.
  arXiv  Detail & Related papers  (2024-03-27T19:52:55Z)
• TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings [35.18238858796925]
  TEncDM is a novel approach to diffusion modeling that operates in the space of pre-trained language model encodings.<n>In our approach, we also employ a transformer-based decoder, specifically designed to incorporate context in the token prediction process.
  arXiv  Detail & Related papers  (2024-02-29T12:25:45Z)
• Enhancing Diffusion Models with Text-Encoder Reinforcement Learning [63.41513909279474]
  Text-to-image diffusion models are typically trained to optimize the log-likelihood objective. Recent research addresses this issue by refining the diffusion U-Net using human rewards through reinforcement learning or direct backpropagation. We demonstrate that by finetuning the text encoder through reinforcement learning, we can enhance the text-image alignment of the results.
  arXiv  Detail & Related papers  (2023-11-27T09:39:45Z)
• DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion [40.246665336996934]
  A spiral interaction architecture for encoder-decoder text diffusion (DiffuSIA) is proposed. DiffuSIA is evaluated on four text generation tasks, including paraphrase, text simplification, question generation, and open-domain dialogue generation.
  arXiv  Detail & Related papers  (2023-05-19T08:30:11Z)
• Speculative Decoding with Big Little Decoder [108.95187338417541]
  Big Little Decoder (BiLD) is a framework that can improve inference efficiency and latency for a wide range of text generation applications. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture.
  arXiv  Detail & Related papers  (2023-02-15T18:55:29Z)
• Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization [108.09419317477986]
  Z-Code++ is a new pre-trained language model optimized for abstractive text summarization. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum.
  arXiv  Detail & Related papers  (2022-08-21T01:00:54Z)
• All in One: Exploring Unified Video-Language Pre-training [44.22059872694995]
  We introduce an end-to-end video-language model, namely textitall-in-one Transformer, that embeds raw video and textual signals into joint representations. The code and pretrained model have been released in https://github.com/showlab/all-in-one.
  arXiv  Detail & Related papers  (2022-03-14T17:06:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.