Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models

• URL: http://arxiv.org/abs/2312.10835v4
• Date: Fri, 5 Apr 2024 08:20:32 GMT
• Title: Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models
• Abstract summary: We investigate the relative quality of samples produced by the teacher text-to-image diffusion model and its distilled student version. We propose an adaptive collaboration between student and teacher diffusion models for effective text-to-image synthesis.
• Score: 24.372192691537897
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation methods have recently shown to be a promising direction to speedup the synthesis of large-scale diffusion models by requiring only a few inference steps. While several powerful distillation methods were recently proposed, the overall quality of student samples is typically lower compared to the teacher ones, which hinders their practical usage. In this work, we investigate the relative quality of samples produced by the teacher text-to-image diffusion model and its distilled student version. As our main empirical finding, we discover that a noticeable portion of student samples exhibit superior fidelity compared to the teacher ones, despite the "approximate" nature of the student. Based on this finding, we propose an adaptive collaboration between student and teacher diffusion models for effective text-to-image synthesis. Specifically, the distilled model produces the initial sample, and then an oracle decides whether it needs further improvements with a slow teacher model. Extensive experiments demonstrate that the designed pipeline surpasses state-of-the-art text-to-image alternatives for various inference budgets in terms of human preference. Furthermore, the proposed approach can be naturally used in popular applications such as text-guided image editing and controllable generation.

Related papers

• IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves [11.54107612856631]
  We show that information contained in teachers can be leveraged to tailor the noise for multi-teacher distillation.<n>We propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation.
  arXiv  Detail & Related papers  (2026-08-25T15:57:52Z)
• Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression [7.859083902013309]
  Diffusion models generate high-quality images through progressive denoising but are computationally intensive due to large model sizes and repeated sampling. We propose Random Conditioning, a novel approach that pairs noised images with randomly selected text conditions to enable efficient, image-free knowledge distillation. Our method allows the student to explore the condition space without generating condition-specific images, resulting in notable improvements in both generation quality and efficiency.
  arXiv  Detail & Related papers  (2025-04-02T05:41:19Z)
• SFDDM: Single-fold Distillation for Diffusion models [4.688721356965585]
  We propose a single-fold distillation algorithm, SFDDM, which can flexibly compress the teacher diffusion model into a student model of any desired step. Experiments on four datasets demonstrate that SFDDM is able to sample high-quality data with steps reduced to as little as approximately 1%.
  arXiv  Detail & Related papers  (2024-05-23T18:11:14Z)
• Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers [120.49126407479717]
  This paper explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos.
  arXiv  Detail & Related papers  (2024-03-12T00:02:03Z)
• Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering [118.53208190209517]
  We propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. Our method can effectively learn the prompts to improve the matches between the input text and the generated images.
  arXiv  Detail & Related papers  (2024-01-12T03:46:29Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)
• Improving Neural Topic Models with Wasserstein Knowledge Distillation [0.8962460460173959]
  We propose a knowledge distillation framework to compress a contextualized topic model without loss in topic quality. Experiments show that the student trained with knowledge distillation achieves topic coherence much higher than that of the original student model.
  arXiv  Detail & Related papers  (2023-03-27T16:07:44Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• Anomaly Detection via Reverse Distillation from One-Class Embedding [2.715884199292287]
  We propose a novel T-S model consisting of a teacher encoder and a student decoder. Instead of receiving raw images directly, the student network takes teacher model's one-class embedding as input. In addition, we introduce a trainable one-class bottleneck embedding module in our T-S model.
  arXiv  Detail & Related papers  (2022-01-26T01:48:37Z)
• Reinforced Multi-Teacher Selection for Knowledge Distillation [54.72886763796232]
  knowledge distillation is a popular method for model compression. Current methods assign a fixed weight to a teacher model in the whole distillation. Most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled.
  arXiv  Detail & Related papers  (2020-12-11T08:56:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.