Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation

• URL: http://arxiv.org/abs/2604.19141v1
• Date: Tue, 21 Apr 2026 06:44:44 GMT
• Title: Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation
• Abstract summary: Diffusion- and flow-based models usually allocate compute uniformly across space.<n>Some regions are easy to denoise, whereas others benefit from more refinement or additional context.<n>We explore patch-level noise scales for image synthesis.
• Score: 22.05381528577716
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion- and flow-based models usually allocate compute uniformly across space, updating all patches with the same timestep and number of function evaluations. While convenient, this ignores the heterogeneity of natural images: some regions are easy to denoise, whereas others benefit from more refinement or additional context. Motivated by this, we explore patch-level noise scales for image synthesis. We find that naively varying timesteps across image tokens performs poorly, as it exposes the model to overly informative training states that do not occur at inference. We therefore introduce a timestep sampler that explicitly controls the maximum patch-level information available during training, and show that moving from global to patch-level timesteps already improves image generation over standard baselines. By further augmenting the model with a lightweight per-patch difficulty head, we enable adaptive samplers that allocate compute dynamically where it is most needed. Combined with noise levels varying over both space and diffusion time, this yields Patch Forcing (PF), a framework that advances easier regions earlier so they can provide context for harder ones. PF achieves superior results on class-conditional ImageNet, remains orthogonal to representation alignment and guidance methods, and scales to text-to-image synthesis. Our results suggest that patch-level denoising schedules provide a promising foundation for adaptive image generation.

Related papers

• Adversarial Learning of Classifier-Free Guidance Schedules [74.70485037604884]
  We learn the guidance schedule as a function of diffusion time, conditioning and the current noisy sample.<n>We frame this as a density ratio estimation problem.<n>A lightweight generator network predicts the optimal, state-dependent guidance scale.
  arXiv  Detail & Related papers  (2026-08-14T07:30:36Z)
• ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models [33.09645476860831]
  We propose ImageRAGTurbo, a novel approach to efficiently finetune few-step diffusion models via retrieval augmentation.<n>Given a text prompt, we retrieve relevant text-image pairs from a database and use them to condition the generation process.<n>Experiments show that our approach produces high-fidelity images without compromising latency compared to existing methods.
  arXiv  Detail & Related papers  (2026-02-13T05:59:57Z)
• TTSnap: Test-Time Scaling of Diffusion Models via Noise-Aware Pruning [53.52543819839442]
  A prominent approach to test-time scaling for text-to-image diffusion models formulates the problem as a search over multiple noise seeds.<n>We propose test-time scaling with noise-aware pruning (TTSnap), a framework that prunes low-quality candidates without fully denoising them.
  arXiv  Detail & Related papers  (2025-11-27T09:14:26Z)
• LOTA: Bit-Planes Guided AI-Generated Image Detection [25.736743931612565]
  GAN and Diffusion models make it more difficult to distinguish AI-generated images from real ones.<n>We introduce an effective bit-planes guided noisy image generation and exploit various image normalization strategies.<n>Our method achieves an accuracy of over 98.2% from GAN to Diffusion and over 99.2% from Diffusion to GAN.
  arXiv  Detail & Related papers  (2025-10-16T02:12:49Z)
• Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation [48.203403471536866]
  We propose a novel framework that allocates distinct timesteps to different pixels and reformulates the pixel-wise denoising process.<n>Our experiments demonstrate that our asynchronous diffusion models can significantly improve text-to-image alignment across diverse prompts.
  arXiv  Detail & Related papers  (2025-10-06T05:45:56Z)
• Halton Scheduler For Masked Generative Image Transformer [51.82285573627426]
  Masked Generative Image Transformers (MaskGIT) have emerged as a scalable and efficient image generation framework.<n>We analyze the sampling objective in MaskGIT, based on the mutual information between tokens.<n>We propose a new sampling strategy based on our Halton scheduler instead of the original Confidence scheduler.
  arXiv  Detail & Related papers  (2025-03-21T12:00:59Z)
• Fast constrained sampling in pre-trained diffusion models [80.99262780028015]
  We propose an algorithm that enables fast, high-quality generation under arbitrary constraints.<n>Our approach produces results that rival or surpass the state-of-the-art training-free inference methods.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• Linear Combinations of Patches are Unreasonably Effective for Single-Image Denoising [5.893124686141782]
  Deep neural networks have revolutionized image denoising in achieving significant accuracy improvements. To alleviate the requirement to learn image priors externally, single-image methods perform denoising solely based on the analysis of the input noisy image. This work investigates the effectiveness of linear combinations of patches for denoising under this constraint.
  arXiv  Detail & Related papers  (2022-12-01T10:52:03Z)
• Differentiable Point-Based Radiance Fields for Efficient View Synthesis [57.56579501055479]
  We propose a differentiable rendering algorithm for efficient novel view synthesis. Our method is up to 300x faster than NeRF in both training and inference. For dynamic scenes, our method trains two orders of magnitude faster than STNeRF and renders at near interactive rate.
  arXiv  Detail & Related papers  (2022-05-28T04:36:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.