Temporal Difference Learning for Diffusion Models

• URL: http://arxiv.org/abs/2606.15048v1
• Date: Sat, 13 Jun 2026 01:34:04 GMT
• Title: Temporal Difference Learning for Diffusion Models
• Abstract summary: We introduce a temporal difference (TD) objective that penalizes inconsistency of the model's multi-step progress along the denoising path.<n>We propose a principled sample-based reweighting method that stabilizes training.
• Score: 19.82560792699663
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models are typically trained with objectives that focus on local denoising targets at individual time steps (or adjacent pairs), which do not enforce consistency between predictions along the denoising trajectory. This lack of cross-time consistency can degrade performance, especially for few-step samplers. We introduce a temporal difference (TD) objective that penalizes inconsistency of the model's multi-step progress along the denoising path. By reformulating the diffusion process as a Markov reward process and casting denoising as a policy evaluation problem in reinforcement learning, we derive a unified TD approach that applies to both discrete- and continuous-time diffusion formulations. We further propose a principled sample-based reweighting method that stabilizes training. Empirically, we show that using our TD training can significantly improve sample quality measured by FID, with stronger advantages when the number of sampling steps is small, highlighting its practical utility under low-computation-budget scenarios. We provide ablation studies to justify our design choices, including pairwise loss reweighting, regularization weight, and one-step stride. Overall, our TD approach can be a general drop-in that enforces cross-time consistency and improves generation quality across different diffusion generative models.

Related papers

• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Memorization Control in Diffusion Models from Denoising-centric Perspective [0.6741942263052466]
  Controlling memorization in diffusion models is critical for applications that require generated data to closely match the training distribution.<n>We show that uniform timestep sampling leads to unequal learning contributions across denoising steps due to differences in signal to noise ratio.<n>We propose a timestep sampling strategy that explicitly controls where learning occurs along the denoising trajectory.
  arXiv  Detail & Related papers  (2026-01-29T07:16:54Z)
• Combating Noisy Labels through Fostering Self- and Neighbor-Consistency [120.4394402099635]
  Label noise is pervasive in various real-world scenarios, posing challenges in supervised deep learning.<n>We propose a noise-robust method named Jo-SNC (textbfJoint sample selection and model regularization based on textbfSelf- and textbfNeighbor-textbfConsistency)<n>We design a self-adaptive, data-driven thresholding scheme to adjust per-class selection thresholds.
  arXiv  Detail & Related papers  (2026-01-19T07:55:29Z)
• Data Unlearning Beyond Uniform Forgetting via Diffusion Time and Frequency Selection [5.350009804371616]
  Data unlearning aims to remove the influence of specific training samples from a trained model without requiring full retraining.<n>We argue that forgetting occurs disproportionately across time and frequency, depending on the model and scenarios.<n>By selectively focusing on specific time-frequency ranges during training, we achieve samples with higher aesthetic quality and lower noise.
  arXiv  Detail & Related papers  (2025-10-20T02:00:12Z)
• Projected Coupled Diffusion for Test-Time Constrained Joint Generation [49.69610867216755]
  We propose Projected Coupled Diffusion (PCD), a novel test-time framework for constrained joint generation.<n>PCD introduces a coupled guidance term into the generative dynamics to encourage coordination between diffusion models and incorporates a projection step at each diffusion step to enforce hard constraints.<n>Our results show improved coupling effects and guaranteed constraint satisfaction without incurring excessive computational costs.
  arXiv  Detail & Related papers  (2025-08-14T11:05:31Z)
• From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster training [16.299508349934754]
  We study the problem of training neural differential equations, or diffusion models, to sample from a Boltzmann distribution without access to target samples.<n>Existing methods for training such models enforce time-reversal of the generative and noising processes.<n>We show that an appropriate choice of coarse time discretization during training allows greatly improved sample efficiency and the use of time-local objectives.
  arXiv  Detail & Related papers  (2025-01-10T18:18:25Z)
• Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning [0.0]
  Fine-tuning foundation models via reinforcement learning (RL) has proven promising for aligning to downstream objectives. We exploit the hierarchical nature of diffusion models (DMs) and train them dynamically at each epoch with a tailored RL method. We show that models trained with HRF achieve better preservation of diversity in downstream tasks, thus enhancing the fine-tuning robustness and at uncompromising mean rewards.
  arXiv  Detail & Related papers  (2024-10-10T19:06:23Z)
• Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training [20.492630610281658]
  Diffusion models learn to denoise data and the trained denoiser is then used to generate new samples from the data distribution. We introduce a new self-supervised training objective that differentiates the levels of noise added to a sample. We show by diverse experiments that the proposed contrastive diffusion training is effective for both sequential and parallel settings.
  arXiv  Detail & Related papers  (2024-07-12T03:03:50Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models [52.1809084559048]
  We propose a novel two-stage divide-and-conquer training strategy termed TDC Training.<n>It groups timesteps based on task similarity and difficulty, assigning highly customized denoising models to each group, thereby enhancing the performance of diffusion models.<n>While two-stage training avoids the need to train each model separately, the total training cost is even lower than training a single unified denoising model.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• Jo-SRC: A Contrastive Approach for Combating Noisy Labels [58.867237220886885]
  We propose a noise-robust approach named Jo-SRC (Joint Sample Selection and Model Regularization based on Consistency) Specifically, we train the network in a contrastive learning manner. Predictions from two different views of each sample are used to estimate its "likelihood" of being clean or out-of-distribution.
  arXiv  Detail & Related papers  (2021-03-24T07:26:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.