DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation

• URL: http://arxiv.org/abs/2607.26811v1
• Date: Wed, 29 Jul 2026 11:58:47 GMT
• Title: DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation
• Abstract summary: We introduce a distributional evaluation protocol that measures precision and coverage between student and teacher distributions.<n>We propose joint distillation, which combines DMD's mode-seeking objective with a Consistency Distillation-based mode-covering constraint.
• Score: 29.992105358526715
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing autoregressive video distillation methods commonly adopt a Distribution Matching Distillation (DMD)-based multi-stage pipeline. However, they typically decouple the initialization and DMD stages -- which then pursue different target distributions -- and judge the intermediate student mainly by visual scores such as VBench. In this paper, we revisit this design from a distributional perspective. Given the mode-seeking nature of the distribution matching loss, a good initialization should match the mode coverage of the target DMD teacher, rather than merely pursuing high quality. To analyze this, we introduce a distributional evaluation protocol that measures precision and coverage between student and teacher distributions in a shared latent space. It exposes differences hidden by visual scores: some initializations reach high precision but low coverage, leading to suboptimal refinement, while mode-covering ones preserve broader support. Furthermore, even when the target distributions are aligned, DMD's reverse-KL objective can still drive the student toward high-probability teacher regions in late training, reducing coverage and diversity. To address this, we propose joint distillation, which combines DMD's mode-seeking objective with a Consistency Distillation-based mode-covering constraint. Experiments show that our method improves generation quality, coverage, and diversity; notably, even with a Wan-1.3B DMD teacher, it outperforms baselines refined with Wan-14B, underscoring the importance of distributional alignment in autoregressive video distillation.

Related papers

• Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher [36.07614733558147]
  We introduce textbfSelf-OPD, a teacher-free OPD framework for flow matching models.<n>Self-OPD branches the deterministic next-state prediction into $K$ SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline.<n>Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.
  arXiv  Detail & Related papers  (2026-08-27T09:34:23Z)
• CoDMD: Copula-aware Distribution Matching Distillation for Fast Video Generation [50.353919095724315]
  Few-step distillation for video diffusion models has attracted significant attention, driven by the urgent demand for efficient deployment in real-world scenarios.<n>We propose Copula-aware DMD (CoDMD), a lightweight relational regularizer that reuses score estimates already produced by the frozen teacher and the online fake model to construct pairwise relation matrices.<n>On the Wan-2.1-T2V model series at 1.3B & 14B scales, CoDMD distills 50-step teachers into 4-step students, achieving an approximate 25$times$ speed-up while attaining VBench scores of 84.46 & 84.87,
  arXiv  Detail & Related papers  (2026-06-20T10:33:14Z)
• SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation [57.297118390628384]
  Distribution Matching Distillation (DMD) is a widely used paradigm for accelerating inference in few-step video diffusion models.<n>We propose textbfScore Gradient Matching Distillation (SGMD).<n>It adopts a fake-score perspective by directly optimizing the fake score toward the teacher, while using teacher stop-gradient Fisher as a stable distribution-matching objective.<n>It substantially improves motion dynamics for 4-step distilled models while preserving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-28T15:50:55Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Continuous-Time Distribution Matching for Few-Step Diffusion Distillation [57.28746398500951]
  We introduce Continuous-Time Distribution Matching (CDM), migrating the DMD framework from discrete anchoring to continuous optimization.<n>CDM achieves this through two continuous-time designs. First, we replace the fixed discrete schedule with a dynamic continuous schedule of random length.<n>Second, we propose a continuous-time alignment objective that performs active off-trajectory matching on latents extrapolated via the student's velocity field.
  arXiv  Detail & Related papers  (2026-05-07T14:56:39Z)
• Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis [33.341277146439275]
  Distribution matching distillation (DMD) aligns a multi-step generator with its few-step counterpart to enable high-quality generation under low inference cost.<n>We propose a role-separated distillation framework that explicitly disentangles the roles of distilled steps.<n>We term this approach Diversity-Preserved DMD (DP-DMD), which, despite its simplicity, preserves sample diversity while maintaining visual quality on par with state-of-the-art methods in extensive text-to-image experiments.
  arXiv  Detail & Related papers  (2026-02-03T05:45:25Z)
• Distillation of Discrete Diffusion by Exact Conditional Distribution Matching [9.460409527892345]
  We propose a simple and principled distillation alternative based on emphconditional distribution matching.<n>We exploit this structure to define distillation objectives that directly match conditional distributions between a pre-trained teacher and a low-NFE student.
  arXiv  Detail & Related papers  (2025-12-15T00:16:10Z)
• Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield [54.328202401611264]
  Diffusion model distillation has emerged as a powerful technique for creating efficient few-step and single-step generators.<n>We show that the primary driver of few-step distillation is not distribution matching, but a previously overlooked component we identify as CFG Augmentation (CA)<n>We propose principled modifications to the distillation process, such as decoupling the noise schedules for the engine and the regularizer, leading to further performance gains.
  arXiv  Detail & Related papers  (2025-11-27T18:24:28Z)
• Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis [65.77083310980896]
  We propose Adrial Distribution Matching (ADM) to align latent predictions between real and fake score estimators for score distillation.<n>Our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time.<n>Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.
  arXiv  Detail & Related papers  (2025-07-24T16:45:05Z)
• Importance Weighted Score Matching for Diffusion Samplers with Enhanced Mode Coverage [16.94974733994214]
  prevailing methods often circumvent the lack of target data by optimizing reverse KL-based objectives.<n>We propose a principled approach for training diffusion-based samplers by directly targeting an objective analogous to the forward KL divergence.<n>Our approach consistently outperforms existing neural samplers across all distributional distance metrics.
  arXiv  Detail & Related papers  (2025-05-26T02:48:26Z)
• DDIL: Diversity Enhancing Diffusion Distillation With Imitation Learning [57.3467234269487]
  Diffusion models excel at generative modeling (e.g., text-to-image) but sampling requires multiple denoising network passes.<n>Progressive distillation or consistency distillation have shown promise by reducing the number of passes.<n>We show that DDIL consistency improves on baseline algorithms of progressive distillation (PD), Latent consistency models (LCM) and Distribution Matching Distillation (DMD2)
  arXiv  Detail & Related papers  (2024-10-15T18:21:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.