Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching

• URL: http://arxiv.org/abs/2609.04283v1
• Date: Thu, 03 Sep 2026 04:46:46 GMT
• Title: Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching
• Abstract summary: We propose a unified, single-stage optimization framework grounded in Distribution Matching.<n>In the standard DM framework, distillation updates the model via a gradient direction that minimizes the gap between the real and fake models.<n>We introduce DM-Align, which derives a complementary gradient direction to guide the model toward human-preferred samples.
• Score: 36.00855152229835
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Aligning video generative models to human preferences heavily relies on Reinforcement Learning (RL), which suffers from extensive computational overhead. Existing workflows typically treat RL and distillation as disconnected stages: applying RL before distillation incurs prohibitive computational costs, whereas applying RL after distillation frequently leads to model collapse. To overcome these limitations, we propose a unified, single-stage optimization framework grounded in Distribution Matching (DM). In the standard DM framework, distillation updates the model via a gradient direction that minimizes the gap between the real and fake models, guiding generations toward clarity and high fidelity. Building upon this, we introduce DM-Align, which derives a complementary gradient direction to guide the model toward human-preferred samples. Inspired by DPO and GRPO, our method leverages the distributional gap -- formulated from either preference pairs or intra-group exploration -- to directly construct this preference-guided gradient. By synergizing these two gradient directions, our approach eliminates the need for multi-step reward evaluation and complex ODE-SDE conversions inherent in traditional RL. Comprehensive experiments across multiple foundational video models demonstrate that this sample-guided framework robustly enhances both distillation quality and preference alignment, consistently outperforming both standalone variants and sequential two-stage pipelines.

Related papers

• Reinforcing Few-step Generators via Reward-Tilted Distribution Matching [34.875805803270524]
  We propose Reward-Tilted Distribution Matching Distillation (RTDMD)<n>We show that minimizing KL divergence to a reward-tilted teacher distribution naturally decomposes into a distribution matching term and a reward term.<n>Experiments on SD3, SD3.5, and FLUX.2 demonstrate that RTDMD establishes new state-of-the-art results across preference, aesthetic, and compositional metrics.
  arXiv  Detail & Related papers  (2026-05-25T17:59:21Z)
• Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models [5.527466367347639]
  We derive a generalized DPO objective that covers both diffusion and flow-matching via a unified reverse-time SDE framework.<n>We propose Linear-DPO, which replaces the aggressive sigmoid-based utility function with a sustained linear utility and incorporates an EMA-updated reference model.
  arXiv  Detail & Related papers  (2026-05-20T12:54:51Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Guiding Distribution Matching Distillation with Gradient-Based Reinforcement Learning [41.982957134224904]
  Diffusion distillation has shown great promise in few-step generation but often sacrifices quality for sampling speed.<n>We propose GDMD, a novel framework that redefines the reward mechanism by prioritizing distillation gradients over raw pixel outputs.<n>Our models outperform the quality of their multi-step teacher and substantially exceed previous DMDR results in GenEval and human-preference metrics.
  arXiv  Detail & Related papers  (2026-04-21T02:57:13Z)
• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization [88.7827307535107]
  Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature.<n>Existing solver-based acceleration methods often face significant image quality degradation under a low-dimensional budget.<n>We propose the Ensemble Parallel Direction solver (dubbed as EPD-EPr), a novel ODE solver that mitigates these errors by incorporating multiple gradient parallel evaluations in each step.
  arXiv  Detail & Related papers  (2025-12-28T05:48:55Z)
• G$^2$RPO: Granular GRPO for Precise Reward in Flow Models [74.21206048155669]
  We propose a novel Granular-GRPO (G$2$RPO) framework that achieves precise and comprehensive reward assessments of sampling directions.<n>We introduce a Multi-Granularity Advantage Integration module that aggregates advantages computed at multiple diffusion scales.<n>Our G$2$RPO significantly outperforms existing flow-based GRPO baselines.
  arXiv  Detail & Related papers  (2025-10-02T12:57:12Z)
• Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis [65.77083310980896]
  We propose Adrial Distribution Matching (ADM) to align latent predictions between real and fake score estimators for score distillation.<n>Our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time.<n>Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.
  arXiv  Detail & Related papers  (2025-07-24T16:45:05Z)
• Adding Additional Control to One-Step Diffusion with Joint Distribution Matching [58.37264951734603]
  JDM is a novel approach that minimizes the reverse KL divergence between image-condition joint distributions.<n>By deriving a tractable upper bound, JDM decouples fidelity learning from condition learning.<n>This asymmetric distillation scheme enables our one-step student to handle controls unknown to the teacher model.
  arXiv  Detail & Related papers  (2025-03-09T15:06:50Z)
• Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimation [18.295352638247362]
  preference labels are typically available only at the terminal of denoising trajectories.<n>We propose Denoised Distribution Estimation that explicitly connects intermediate steps to the terminal denoised distribution.<n>Our approach achieves superior performance, both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2024-11-22T11:45:33Z)
• Distilling Diffusion Models into Conditional GANs [90.76040478677609]
  We distill a complex multistep diffusion model into a single-step conditional GAN student model. For efficient regression loss, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model's latent space. We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models.
  arXiv  Detail & Related papers  (2024-05-09T17:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.