Scaling Down Text Encoders of Text-to-Image Diffusion Models

• URL: http://arxiv.org/abs/2503.19897v1
• Date: Tue, 25 Mar 2025 17:55:20 GMT
• Title: Scaling Down Text Encoders of Text-to-Image Diffusion Models
• Authors: Lifu Wang, Daqing Liu, Xinchen Liu, Xiaodong He,
• Abstract summary: Text encoders in diffusion models have rapidly evolved, transitioning from CLIP to T5-XXL.<n>We employ vision-based knowledge distillation to train a series of T5 encoder models.<n>Our results demonstrate the scaling down pattern that the distilled T5-base model can generate images of comparable quality to those produced by T5-XXL.
• Score: 24.751226627178475
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text encoders in diffusion models have rapidly evolved, transitioning from CLIP to T5-XXL. Although this evolution has significantly enhanced the models' ability to understand complex prompts and generate text, it also leads to a substantial increase in the number of parameters. Despite T5 series encoders being trained on the C4 natural language corpus, which includes a significant amount of non-visual data, diffusion models with T5 encoder do not respond to those non-visual prompts, indicating redundancy in representational power. Therefore, it raises an important question: "Do we really need such a large text encoder?" In pursuit of an answer, we employ vision-based knowledge distillation to train a series of T5 encoder models. To fully inherit its capabilities, we constructed our dataset based on three criteria: image quality, semantic understanding, and text-rendering. Our results demonstrate the scaling down pattern that the distilled T5-base model can generate images of comparable quality to those produced by T5-XXL, while being 50 times smaller in size. This reduction in model size significantly lowers the GPU requirements for running state-of-the-art models such as FLUX and SD3, making high-quality text-to-image generation more accessible.

Related papers

• Decoder-Only LLMs are Better Controllers for Diffusion Models [63.22040456010123]
  We propose to enhance text-to-image diffusion models by borrowing the strength of semantic understanding from large language models.<n>Our adapter module is superior to the stat-of-the-art models in terms of text-to-image generation quality and reliability.
  arXiv  Detail & Related papers  (2025-02-06T12:17:35Z)
• Mimir: Improving Video Diffusion Models for Precise Text Understanding [53.72393225042688]
  Text serves as the key control signal in video generation due to its narrative nature. The recent success of large language models (LLMs) showcases the power of decoder-only transformers. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser.
  arXiv  Detail & Related papers  (2024-12-04T07:26:44Z)
• TextCraftor: Your Text Encoder Can be Image Quality Controller [65.27457900325462]
  Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have revolutionized the field of content generation. We propose a proposed fine-tuning approach, TextCraftor, to enhance the performance of text-to-image diffusion models.
  arXiv  Detail & Related papers  (2024-03-27T19:52:55Z)
• Paragraph-to-Image Generation with Information-Enriched Diffusion Model [67.9265336953134]
  ParaDiffusion is an information-enriched diffusion model for paragraph-to-image generation task. It delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation. The code and dataset will be released to foster community research on long-text alignment.
  arXiv  Detail & Related papers  (2023-11-24T05:17:01Z)
• The Five-Dollar Model: Generating Game Maps and Sprites from Sentence Embeddings [3.620115940532283]
  The five-dollar model is a lightweight text-to-image generative architecture that generates low dimensional images from an encoded text prompt. We apply this model to three small datasets: pixel art video game maps, video game sprite images, and down-scaled emoji images. We evaluate our models performance using cosine similarity score between text-image pairs generated by the CLIP VIT-B/32 model.
  arXiv  Detail & Related papers  (2023-08-08T05:16:51Z)
• Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization [108.09419317477986]
  Z-Code++ is a new pre-trained language model optimized for abstractive text summarization. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum.
  arXiv  Detail & Related papers  (2022-08-21T01:00:54Z)
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding [53.170767750244366]
  Imagen is a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models.
  arXiv  Detail & Related papers  (2022-05-23T17:42:53Z)
• EncT5: Fine-tuning T5 Encoder for Non-autoregressive Tasks [9.141586109808895]
  We study fine-tuning pre-trained encoder-decoder models such as T5. Our experimental results show that textbfEncT5 with less than half of the parameters of T5 performs similarly to T5 models on GLUE benchmark.
  arXiv  Detail & Related papers  (2021-10-16T00:50:08Z)
• Attention Is Indeed All You Need: Semantically Attention-Guided Decoding for Data-to-Text NLG [0.913755431537592]
  We propose a novel decoding method that extracts interpretable information from encoder-decoder models' cross-attention. We show on three datasets its ability to dramatically reduce semantic errors in the generated outputs.
  arXiv  Detail & Related papers  (2021-09-15T01:42:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.