Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting

• URL: http://arxiv.org/abs/2607.10853v1
• Date: Sun, 12 Jul 2026 17:32:29 GMT
• Title: Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting
• Abstract summary: We introduce variance-corrective time shifting, a training-free fix that queries the network at a shifted timestep and scales the resulting score by $$, canceling the variance inflation while preserving the mode reweighting.<n>We demonstrate consistent gains at minimal cost to sample quality and condition fidelity across DiT, Stable Diffusion and Motion Diffusion models.
• Score: 57.75750707482832
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models faithfully reproduce their training distribution, but also inherit its imbalances and leave rare or under-represented modes hard to reach. A natural inference-time remedy is to sample from the high-temperature target $p^{(γ)}_0(x) \propto p_0(x)^γ$ for $0 < γ< 1$, which flattens dominant modes and lifts rare ones. However, naive score scaling while correctly reweighting modes also inflates the per-mode variance, breaking the reverse diffusion process and degrading sample quality. We introduce variance-corrective time shifting, a training-free fix that queries the network at a shifted timestep and scales the resulting score by $γ$, canceling the variance inflation while preserving the mode reweighting. The correction turns simple temperature sampling into a practical diversity knob for pretrained diffusion and flow-matching backbones with no retraining, and we demonstrate consistent gains at minimal cost to sample quality and condition fidelity across DiT, Stable Diffusion and Motion Diffusion models. We further show that the timing of the temperature intervention enables coarse-to-fine control: high-noise stages drive compositional diversity across modes, while low-noise stages drive local appearance variation under a fixed composition.

Related papers

• Emergence of Distortions in High-Dimensional Guided Diffusion Models [11.774563966512707]
  We formalize the phenomenon of generative distortion defined as the mismatch between the CFG-induced sampling and the true conditional distribution.<n>We show that standard CFG schedules are incapable of preventing variance shrinkage.<n>We propose a theoretically motivated guidance schedule featuring a negative-guidance window, which mitigates loss of diversity while preserving class separability.
  arXiv  Detail & Related papers  (2026-01-31T13:19:45Z)
• Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models [14.963168108932761]
  We argue that hot diffusion fails to exploit the strong correlation between high-frequency image detail and low-frequency structures.<n>We propose a unified Blur-Noise Mixture Diffusion Model (BNMD) to control blurring and noise jointly.
  arXiv  Detail & Related papers  (2025-11-21T02:37:09Z)
• Progressive Inference-Time Annealing of Diffusion Models for Sampling from Boltzmann Densities [93.13866975467549]
  We propose Progressive Inference-Time Annealing (PITA) to learn diffusion-based samplers.<n>PITA combines two complementary techniques: Annealing of the Boltzmann distribution and Diffusion smoothing.<n>It enables equilibrium sampling of N-body particle systems, Alanine Dipeptide, and tripeptides in Cartesian coordinates.
  arXiv  Detail & Related papers  (2025-06-19T17:14:22Z)
• FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering [0.0]
  Multi-modal and high-dimensional posteriors present significant challenges for variational inference.<n>We introduce FlowVAT, a conditional tempering approach for normalizing flow variational inference.<n>Our method tempers both the base and target distributions simultaneously, maintaining affine-invariance under tempering.
  arXiv  Detail & Related papers  (2025-05-15T16:20:36Z)
• Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation [21.64693536216534]
  Self-consistency improves reasoning by aggregating diverse samples.<n>We show that decoding temperature governs randomness and actively shapes the latent answer distribution.<n>We propose a confidence-driven mechanism that dynamically calibrates temperature.
  arXiv  Detail & Related papers  (2025-02-27T07:07:40Z)
• Repulsive Latent Score Distillation for Solving Inverse Problems [31.255943277671893]
  Score Distillation Sampling (SDS) has been pivotal for leveraging pre-trained diffusion models in downstream tasks such as inverse problems. We introduce a novel variational framework for posterior sampling to address mode collapse and latent space inversion. We extend this framework with an augmented variational distribution that disentangles the latent and data.
  arXiv  Detail & Related papers  (2024-06-24T14:43:02Z)
• Boosting Diffusion Models with Moving Average Sampling in Frequency Domain [101.43824674873508]
  Diffusion models rely on the current sample to denoise the next one, possibly resulting in denoising instability. In this paper, we reinterpret the iterative denoising process as model optimization and leverage a moving average mechanism to ensemble all the prior samples. We name the complete approach "Moving Average Sampling in Frequency domain (MASF)"
  arXiv  Detail & Related papers  (2024-03-26T16:57:55Z)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models [101.831766524264]
  Inverse tasks can be formulated as inferring a posterior distribution over data. This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. We propose a variational approach that by design seeks to approximate the true posterior distribution.
  arXiv  Detail & Related papers  (2023-05-07T23:00:47Z)
• ShiftDDPMs: Exploring Conditional Diffusion Models by Shifting Diffusion Trajectories [144.03939123870416]
  We propose a novel conditional diffusion model by introducing conditions into the forward process. We use extra latent space to allocate an exclusive diffusion trajectory for each condition based on some shifting rules. We formulate our method, which we call textbfShiftDDPMs, and provide a unified point of view on existing related methods.
  arXiv  Detail & Related papers  (2023-02-05T12:48:21Z)
• Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors [64.24948495708337]
  We introduce a new method that brings predicted samples to the training data manifold using a pretrained unconditional diffusion model. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks.
  arXiv  Detail & Related papers  (2022-12-14T17:26:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.