On-Policy Self-Distillation in Diffusion Models

• URL: http://arxiv.org/abs/2608.24646v1
• Date: Tue, 25 Aug 2026 14:55:24 GMT
• Title: On-Policy Self-Distillation in Diffusion Models
• Abstract summary: Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change.<n>We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries.<n>Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators.
• Score: 89.32656931795293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change. We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries. At each outer iteration, a frozen behavior policy generates trajectories and supplies query states and anchors. Reward gradients construct bounded positive and negative targets around each anchor. The trainable policy fits these targets as detached supervision through finite fitting before an exponential moving average update refreshes the behavior policy. This setup lets us measure target construction and finite realization separately. Controlled same-query experiments show that larger target-construction gains do not necessarily translate into larger realized gains after a single fitting update. Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators. It outperforms the strongest competing method by up to 44.0% and reduces training GPU-hours relative to DiffusionNFT by 40% on SD 3.5-M and 63% on Z-Image-Turbo. These results support on-policy self-distillation as an efficient and analyzable approach to diffusion post-training by converting image-level reward guidance into explicit and continually refreshed intermediate supervision, thereby opening a path toward more efficient and diagnosable alignment.

Related papers

• I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization [5.536267204193318]
  Group Relative Policy Optimization learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect.<n> Privileged self-distillation can fill this gap with dense token supervision, yet applying it throughout training creates a different failure mode.<n>We introduce I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), which treats teacher reliance as capability-dependent.
  arXiv  Detail & Related papers  (2026-08-13T08:37:24Z)
• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training [51.18425726646089]
  We propose textscSURE, a unified latent-space framework for image and video diffusion models.<n>It learns reward distributions and directly uses their reliability to guide dense post-training.<n>textscSURE-REFL achieves the highest VBench quality, semantic, and total scores among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-06T14:55:42Z)
• Physics-Guided Policy Optimization with Self-Distillation [13.212487988108592]
  Self-distilled policy optimization (SDPO) has become a popular paradigm for LLM post-training.<n>We propose Physics-Guided Policy Optimization (PGPO), which introduces an information-modulated step-size multiplier.<n>We show that this modulation preserves the order-1 weak-approximation guarantees of vanilla SGD, and incurs negligible overhead per iteration.
  arXiv  Detail & Related papers  (2026-06-02T13:20:39Z)
• Flow-OPD: On-Policy Distillation for Flow Matching Models [42.290892456464896]
  Flow-OPD is a post-training framework that integrates on-policy distillation into Flow Matching models.<n>It raises the GenEval score from 63 to 92 and the OCR accuracy from 59 to 94, yielding an overall improvement of roughly 10 points over vanilla GRPO.<n>These results establish Flow-OPD as a scalable alignment paradigm for building generalist text-to-image models.
  arXiv  Detail & Related papers  (2026-05-08T17:50:15Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [79.88256756334327]
  Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision.<n>We propose Sample-Routed Policy Optimization (SRPO)<n>SRPO routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction.
  arXiv  Detail & Related papers  (2026-04-02T17:29:18Z)
• Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies [31.52910494173408]
  Diffusion-based robot navigation policies can generate multi-modal trajectories directly from the robot's visual observations.<n>We propose a reinforcement learning framework tailored for diffusion-based navigation.<n>Our approach improves the Success Rate from 52.0% to 58.7% and SPL from 0.49 to 0.54 on unseen scenes, while reducing collision frequency.
  arXiv  Detail & Related papers  (2026-03-13T10:14:32Z)
• Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design [45.80068602880684]
  We provide a systematic analysis of the RL design space by disentangling three factors: policy-gradient objectives, likelihood estimators, and rollout sampling schemes.<n>We show that adopting an evidence lower bound (ELBO) based model likelihood estimator, computed only from the final generated sample, is the dominant factor enabling effective, efficient, and stable RL optimization.
  arXiv  Detail & Related papers  (2026-02-04T15:36:42Z)
• DiffusionNFT: Online Diffusion Reinforcement with Forward Process [99.94852379720153]
  Diffusion Negative-aware FineTuning (DiffusionNFT) is a new online RL paradigm that optimize diffusion models directly on the forward process via flow matching.<n>DiffusionNFT is up to $25times$ more efficient than FlowGRPO in head-to-head comparisons, while being CFG-free.
  arXiv  Detail & Related papers  (2025-09-19T16:09:33Z)
• Manifold-Aware Self-Training for Unsupervised Domain Adaptation on Regressing 6D Object Pose [69.14556386954325]
  Domain gap between synthetic and real data in visual regression is bridged in this paper via global feature alignment and local refinement. Our method incorporates an explicit self-supervised manifold regularization, revealing consistent cumulative target dependency across domains. Learning unified implicit neural functions to estimate relative direction and distance of targets to their nearest class bins aims to refine target classification predictions.
  arXiv  Detail & Related papers  (2023-05-18T08:42:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.