CoDe: Blockwise Control for Denoising Diffusion Models

• URL: http://arxiv.org/abs/2502.00968v2
• Date: Sat, 03 May 2025 14:56:26 GMT
• Title: CoDe: Blockwise Control for Denoising Diffusion Models
• Authors: Anuj Singh, Sayak Mukherjee, Ahmad Beirami, Hadi Jamali-Rad,
• Abstract summary: Aligning diffusion models to downstream tasks often requires finetuning new models or gradient-based guidance at inference time.<n>In this work, we explore a simple inference-time gradient-free guidance approach, called controlled denoising (CoDe)<n>CoDe is a blockwise sampling method applied during intermediate denoising steps, allowing for alignment with downstream rewards.
• Score: 9.235074675079767
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aligning diffusion models to downstream tasks often requires finetuning new models or gradient-based guidance at inference time to enable sampling from the reward-tilted posterior. In this work, we explore a simple inference-time gradient-free guidance approach, called controlled denoising (CoDe), that circumvents the need for differentiable guidance functions and model finetuning. CoDe is a blockwise sampling method applied during intermediate denoising steps, allowing for alignment with downstream rewards. Our experiments demonstrate that, despite its simplicity, CoDe offers a favorable trade-off between reward alignment, prompt instruction following, and inference cost, achieving a competitive performance against the state-of-the-art baselines. Our code is available at: https://github.com/anujinho/code.

Related papers

• Exploiting the Exact Denoising Posterior Score in Training-Free Guidance of Diffusion Models [0.0]
  A popular class of methods, based on Diffusion Posterior Sampling (DPS), attempts to approximate the intractable posterior score function directly.<n>We present a novel expression for the exact posterior score for purely denoising tasks that is tractable in terms of the unconditional score function.<n>We demonstrate that these step sizes are transferable to related inverse problems such as colorization, random inpainting, and super resolution.
  arXiv  Detail & Related papers  (2025-06-16T15:43:28Z)
• Training-free Diffusion Model Alignment with Sampling Demons [15.400553977713914]
  We propose an optimization approach, dubbed Demon, to guide the denoising process at inference time without backpropagation through reward functions or model retraining. Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through optimization. To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models.
  arXiv  Detail & Related papers  (2024-10-08T07:33:49Z)
• ConsistencyDet: A Robust Object Detector with a Denoising Paradigm of Consistency Model [28.193325656555803]
  We introduce a novel framework designed to articulate object detection as a denoising diffusion process. This framework, termed ConsistencyDet, leverages an innovative denoising concept known as the Consistency Model. We show that ConsistencyDet surpasses other leading-edge detectors in performance metrics.
  arXiv  Detail & Related papers  (2024-04-11T14:08:45Z)
• Image Restoration by Denoising Diffusion Models with Iteratively Preconditioned Guidance [9.975341265604577]
  Training deep neural networks has become a common approach for addressing image restoration problems. In low-noise settings, guidance that is based on back-projection (BP) has been shown to be a promising strategy. We propose a novel guidance technique, based on preconditioning that allows traversing from BP-based guidance to at least squares based guidance.
  arXiv  Detail & Related papers  (2023-12-27T10:57:03Z)
• Observation-Guided Diffusion Probabilistic Models [41.749374023639156]
  We propose a novel diffusion-based image generation method called the observation-guided diffusion probabilistic model (OGDM) Our approach reestablishes the training objective by integrating the guidance of the observation process with the Markov chain. We demonstrate the effectiveness of our training algorithm using diverse inference techniques on strong diffusion model baselines.
  arXiv  Detail & Related papers  (2023-10-06T06:29:06Z)
• DeNoising-MOT: Towards Multiple Object Tracking with Severe Occlusions [52.63323657077447]
  We propose DNMOT, an end-to-end trainable DeNoising Transformer for multiple object tracking. Specifically, we augment the trajectory with noises during training and make our model learn the denoising process in an encoder-decoder architecture. We conduct extensive experiments on the MOT17, MOT20, and DanceTrack datasets, and the experimental results show that our method outperforms previous state-of-the-art methods by a clear margin.
  arXiv  Detail & Related papers  (2023-09-09T04:40:01Z)
• DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration [38.4461170690033]
  We propose a novel fine-tuning framework, namely distribution regularization with semantic calibration (DR-Tune) DR-Tune employs distribution regularization by enforcing the downstream task head to decrease its classification error on the pretrained feature distribution. To alleviate the interference by semantic drift, we develop the semantic calibration (SC) module.
  arXiv  Detail & Related papers  (2023-08-23T10:59:20Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• Reflected Diffusion Models [93.26107023470979]
  We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data. Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
  arXiv  Detail & Related papers  (2023-04-10T17:54:38Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Improving Diffusion Models for Inverse Problems using Manifold Constraints [55.91148172752894]
  We show that current solvers throw the sample path off the data manifold, and hence the error accumulates. To address this, we propose an additional correction term inspired by the manifold constraint. We show that our method is superior to the previous methods both theoretically and empirically.
  arXiv  Detail & Related papers  (2022-06-02T09:06:10Z)
• Diffusion-Based Representation Learning [65.55681678004038]
  We augment the denoising score matching framework to enable representation learning without any supervised signal. In contrast, the introduced diffusion-based representation learning relies on a new formulation of the denoising score matching objective. Using the same approach, we propose to learn an infinite-dimensional latent code that achieves improvements of state-of-the-art models on semi-supervised image classification.
  arXiv  Detail & Related papers  (2021-05-29T09:26:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.