Learning Few-Step Diffusion Models by Trajectory Distribution Matching

• URL: http://arxiv.org/abs/2503.06674v2
• Date: Wed, 12 Mar 2025 12:25:18 GMT
• Title: Learning Few-Step Diffusion Models by Trajectory Distribution Matching
• Authors: Yihong Luo, Tianyang Hu, Jiacheng Sun, Yujun Cai, Jing Tang,
• Abstract summary: Trajectory Distribution Matching (TDM) is a unified distillation paradigm that combines the strengths of distribution and trajectory matching.<n>We develop a sampling-steps-aware objective that decouples learning targets across different steps, enabling more adjustable sampling.<n>Our model, TDM, outperforms existing methods on various backbones, delivering superior quality and significantly reduced training costs.
• Score: 18.229753357571116
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accelerating diffusion model sampling is crucial for efficient AIGC deployment. While diffusion distillation methods -- based on distribution matching and trajectory matching -- reduce sampling to as few as one step, they fall short on complex tasks like text-to-image generation. Few-step generation offers a better balance between speed and quality, but existing approaches face a persistent trade-off: distribution matching lacks flexibility for multi-step sampling, while trajectory matching often yields suboptimal image quality. To bridge this gap, we propose learning few-step diffusion models by Trajectory Distribution Matching (TDM), a unified distillation paradigm that combines the strengths of distribution and trajectory matching. Our method introduces a data-free score distillation objective, aligning the student's trajectory with the teacher's at the distribution level. Further, we develop a sampling-steps-aware objective that decouples learning targets across different steps, enabling more adjustable sampling. This approach supports both deterministic sampling for superior image quality and flexible multi-step adaptation, achieving state-of-the-art performance with remarkable efficiency. Our model, TDM, outperforms existing methods on various backbones, such as SDXL and PixArt-$\alpha$, delivering superior quality and significantly reduced training costs. In particular, our method distills PixArt-$\alpha$ into a 4-step generator that outperforms its teacher on real user preference at 1024 resolution. This is accomplished with 500 iterations and 2 A800 hours -- a mere 0.01% of the teacher's training cost. In addition, our proposed TDM can be extended to accelerate text-to-video diffusion. Notably, TDM can outperform its teacher model (CogVideoX-2B) by using only 4 NFE on VBench, improving the total score from 80.91 to 81.65. Project page: https://tdm-t2x.github.io/

Related papers

• DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization [50.30051934609654]
  We introduce a distillation method that combines variational score distillation and consistency distillation to achieve few-step video generation. Our method demonstrates state-of-the-art performance in few-step generation for 10-second videos (128 frames at 12 FPS) One-step distillation accelerates the teacher model's diffusion sampling by up to 278.6 times, enabling near real-time generation.
  arXiv  Detail & Related papers  (2024-12-20T09:07:36Z)
• Accelerating Video Diffusion Models via Distribution Matching [26.475459912686986]
  This work introduces a novel framework for diffusion distillation and distribution matching. Our approach focuses on distilling pre-trained diffusion models into a more efficient few-step generator. By leveraging a combination of video GAN loss and a novel 2D score distribution matching loss, we demonstrate the potential to generate high-quality video frames.
  arXiv  Detail & Related papers  (2024-12-08T11:36:32Z)
• Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator [81.81748032199813]
  Diffusion models have demonstrated excellent performance for real-world image super-resolution (Real-ISR) We propose a new One-Step textbfDiffusion model with a larger-scale textbfDiscriminator for SR. Our discriminator is able to distill noisy features from any time step of diffusion models in the latent space.
  arXiv  Detail & Related papers  (2024-10-05T16:41:36Z)
• TLCM: Training-efficient Latent Consistency Model for Image Generation with 2-8 Steps [12.395969703425648]
  Distilling latent diffusion models (LDMs) into ones that are fast to sample from is attracting growing research interest. This paper proposes a novel training-efficient Latent Consistency Model (TLCM) to overcome these challenges. With just 70 training hours on an A100 GPU, a 3-step TLCM distilled from SDXL achieves an impressive CLIP Score of 33.68 and an Aesthetic Score of 5.97 on the MSCOCO-2017 5K benchmark.
  arXiv  Detail & Related papers  (2024-06-09T12:55:50Z)
• Diffusion Models Are Innate One-Step Generators [2.3359837623080613]
  Diffusion Models (DMs) can generate remarkable high-quality results. DMs' layers are differentially activated at different time steps, leading to an inherent capability to generate images in a single step. Our method achieves the SOTA results on CIFAR-10, AFHQv2 64x64 (FID 1.23), FFHQ 64x64 (FID 0.85) and ImageNet 64x64 (FID 1.16) with great efficiency.
  arXiv  Detail & Related papers  (2024-05-31T11:14:12Z)
• EM Distillation for One-step Diffusion Models [65.57766773137068]
  We propose a maximum likelihood-based approach that distills a diffusion model to a one-step generator model with minimal loss of quality.<n>We develop a reparametrized sampling scheme and a noise cancellation technique that together stabilizes the distillation process.
  arXiv  Detail & Related papers  (2024-05-27T05:55:22Z)
• Improved Distribution Matching Distillation for Fast Image Synthesis [54.72356560597428]
  We introduce DMD2, a set of techniques that lift this limitation and improve DMD training. First, we eliminate the regression loss and the need for expensive dataset construction. Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images.
  arXiv  Detail & Related papers  (2024-05-23T17:59:49Z)
• Directly Denoising Diffusion Models [6.109141407163027]
  We present Directly Denoising Diffusion Model (DDDM), a simple and generic approach for generating realistic images with few-step sampling. Our model achieves FID scores of 2.57 and 2.33 on CIFAR-10 in one-step and two-step sampling respectively, surpassing those obtained from GANs and distillation-based models. For ImageNet 64x64, our approach stands as a competitive contender against leading models.
  arXiv  Detail & Related papers  (2024-05-22T11:20:32Z)
• One-Step Diffusion Distillation via Deep Equilibrium Models [64.11782639697883]
  We introduce a simple yet effective means of distilling diffusion models directly from initial noise to the resulting image. Our method enables fully offline training with just noise/image pairs from the diffusion model. We demonstrate that the DEQ architecture is crucial to this capability, as GET matches a $5times$ larger ViT in terms of FID scores.
  arXiv  Detail & Related papers  (2023-12-12T07:28:40Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.