How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models

• URL: http://arxiv.org/abs/2504.03072v1
• Date: Thu, 03 Apr 2025 22:49:56 GMT
• Title: How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models
• Authors: Pascal Chang, Jingwei Tang, Markus Gross, Vinicius C. Azevedo,
• Abstract summary: We propose a novel method for preserving temporal correlations in a sequence of noise samples.<n>$int$-noise (integral noise) reinterprets individual noise samples as a continuously integrated noise field.<n>$int$-noise can be used for a variety of tasks, such as video restoration, surrogate rendering, and conditional video generation.
• Score: 7.89220773721457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video editing and generation methods often rely on pre-trained image-based diffusion models. During the diffusion process, however, the reliance on rudimentary noise sampling techniques that do not preserve correlations present in subsequent frames of a video is detrimental to the quality of the results. This either produces high-frequency flickering, or texture-sticking artifacts that are not amenable to post-processing. With this in mind, we propose a novel method for preserving temporal correlations in a sequence of noise samples. This approach is materialized by a novel noise representation, dubbed $\int$-noise (integral noise), that reinterprets individual noise samples as a continuously integrated noise field: pixel values do not represent discrete values, but are rather the integral of an underlying infinite-resolution noise over the pixel area. Additionally, we propose a carefully tailored transport method that uses $\int$-noise to accurately advect noise samples over a sequence of frames, maximizing the correlation between different frames while also preserving the noise properties. Our results demonstrate that the proposed $\int$-noise can be used for a variety of tasks, such as video restoration, surrogate rendering, and conditional video generation. See https://warpyournoise.github.io/ for video results.

Related papers

• The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise [92.53724347718173]
  Diffusion models have achieved remarkable success in text-to-image generation tasks. We identify specific regions within the initial noise image, termed trigger patches, that play a key role for object generation in the resulting images.
  arXiv  Detail & Related papers  (2024-06-04T05:06:00Z)
• NoiseDiffusion: Correcting Noise for Image Interpolation with Diffusion Models beyond Spherical Linear Interpolation [86.7260950382448]
  We propose a novel approach to correct noise for image validity, NoiseDiffusion. NoiseDiffusion performs within the noisy image space and injects raw images into these noisy counterparts to address the challenge of information loss.
  arXiv  Detail & Related papers  (2024-03-13T12:32:25Z)
• Blue noise for diffusion models [50.99852321110366]
  We introduce a novel and general class of diffusion models taking correlated noise within and across images into account. Our framework allows introducing correlation across images within a single mini-batch to improve gradient flow. We perform both qualitative and quantitative evaluations on a variety of datasets using our method.
  arXiv  Detail & Related papers  (2024-02-07T14:59:25Z)
• A Generative Model for Digital Camera Noise Synthesis [12.236112464800403]
  We propose an effective generative model which utilizes clean features as guidance followed by noise injections into the network. Specifically, our generator follows a UNet-like structure with skip connections but without downsampling and upsampling layers. We show that our proposed approach outperforms existing methods for synthesizing camera noise.
  arXiv  Detail & Related papers  (2023-03-16T10:17:33Z)
• VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation [88.49030739715701]
  This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation.
  arXiv  Detail & Related papers  (2023-03-15T02:16:39Z)
• Learning to Generate Realistic Noisy Images via Pixel-level Noise-aware Adversarial Training [50.018580462619425]
  We propose a novel framework, namely Pixel-level Noise-aware Generative Adrial Network (PNGAN) PNGAN employs a pre-trained real denoiser to map the fake and real noisy images into a nearly noise-free solution space. For better noise fitting, we present an efficient architecture Simple Multi-versa-scale Network (SMNet) as the generator.
  arXiv  Detail & Related papers  (2022-04-06T14:09:02Z)
• Noise2Score: Tweedie's Approach to Self-Supervised Image Denoising without Clean Images [35.41467558264341]
  We present a novel approach, called Noise2Score, which reveals a missing link in order to unite different approaches. Specifically, we show that image denoising problems without clean images can be addressed by finding the mode of the posterior distribution. Our method then uses the recent finding that the score function can be stably estimated from the noisy images using the amortized residual denoising autoencoder.
  arXiv  Detail & Related papers  (2021-06-13T14:41:09Z)
• Adaptive noise imitation for image denoising [58.21456707617451]
  We develop a new textbfadaptive noise imitation (ADANI) algorithm that can synthesize noisy data from naturally noisy images. To produce realistic noise, a noise generator takes unpaired noisy/clean images as input, where the noisy image is a guide for noise generation. Coupling the noisy data output from ADANI with the corresponding ground-truth, a denoising CNN is then trained in a fully-supervised manner.
  arXiv  Detail & Related papers  (2020-11-30T02:49:36Z)
• Handling noise in image deblurring via joint learning [0.3407858371718068]
  Many blind deblurring methods assume blurred images are noise-free and perform unsatisfactorily on the blurry images with noise. We propose a cascaded framework consisting of a denoiser subnetwork and a deblurring subnetwork. Joint learning reduces the effect of the residual noise after denoising on deblurring, hence improves the robustness of deblurring to heavy noise.
  arXiv  Detail & Related papers  (2020-01-27T12:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.