Elucidating the representation of images within an unconditional diffusion model denoiser

• URL: http://arxiv.org/abs/2506.01912v1
• Date: Mon, 02 Jun 2025 17:33:34 GMT
• Title: Elucidating the representation of images within an unconditional diffusion model denoiser
• Authors: Zahra Kadkhodaie, Stéphane Mallat, Eero Simoncelli,
• Abstract summary: Generative diffusion models learn probability densities over diverse image datasets by estimating the score with a neural network trained to remove noise.<n>Here, we examine a UNet trained for denoising on the ImageNet dataset, to better understand its internal representation and computation of the score.<n>We show that the middle block of the UNet decomposes individual images into sparse subsets of active channels, and that the vector of spatial averages of these channels can provide a nonlinear representation of the underlying clean images.
• Score: 10.853652149844999
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative diffusion models learn probability densities over diverse image datasets by estimating the score with a neural network trained to remove noise. Despite their remarkable success in generating high-quality images, the internal mechanisms of the underlying score networks are not well understood. Here, we examine a UNet trained for denoising on the ImageNet dataset, to better understand its internal representation and computation of the score. We show that the middle block of the UNet decomposes individual images into sparse subsets of active channels, and that the vector of spatial averages of these channels can provide a nonlinear representation of the underlying clean images. We develop a novel algorithm for stochastic reconstruction of images from this representation and demonstrate that it recovers a sample from a set of images defined by a target image representation. We then study the properties of the representation and demonstrate that Euclidean distances in the latent space correspond to distances between conditional densities induced by representations as well as semantic similarities in the image space. Applying a clustering algorithm in the representation space yields groups of images that share both fine details (e.g., specialized features, textured regions, small objects), as well as global structure, but are only partially aligned with object identities. Thus, we show for the first time that a network trained solely on denoising contains a rich and accessible sparse representation of images.

Related papers

• UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation [64.01742988773745]
  An increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. We exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures.
  arXiv  Detail & Related papers  (2024-10-13T16:34:46Z)
• EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models [52.3015009878545]
  We develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. Our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images.
  arXiv  Detail & Related papers  (2024-01-22T07:34:06Z)
• Image segmentation with traveling waves in an exactly solvable recurrent neural network [71.74150501418039]
  We show that a recurrent neural network can effectively divide an image into groups according to a scene's structural characteristics. We present a precise description of the mechanism underlying object segmentation in this network. We then demonstrate a simple algorithm for object segmentation that generalizes across inputs ranging from simple geometric objects in grayscale images to natural images.
  arXiv  Detail & Related papers  (2023-11-28T16:46:44Z)
• Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation [24.436957604430678]
  We develop a neural network architecture which, trained in an unsupervised manner as a denoising diffusion model, simultaneously learns to both generate and segment images. Experiments demonstrate that our model achieves accurate unsupervised image segmentation and high-quality synthetic image generation across multiple datasets.
  arXiv  Detail & Related papers  (2023-09-27T15:32:46Z)
• Learning to Annotate Part Segmentation with Gradient Matching [58.100715754135685]
  This paper focuses on tackling semi-supervised part segmentation tasks by generating high-quality images with a pre-trained GAN. In particular, we formulate the annotator learning as a learning-to-learn problem. We show that our method can learn annotators from a broad range of labelled images including real images, generated images, and even analytically rendered images.
  arXiv  Detail & Related papers  (2022-11-06T01:29:22Z)
• Decoupled Mixup for Generalized Visual Recognition [71.13734761715472]
  We propose a novel "Decoupled-Mixup" method to train CNN models for visual recognition. Our method decouples each image into discriminative and noise-prone regions, and then heterogeneously combines these regions to train CNN models. Experiment results show the high generalization performance of our method on testing data that are composed of unseen contexts.
  arXiv  Detail & Related papers  (2022-10-26T15:21:39Z)
• Deep Semantic Statistics Matching (D2SM) Denoising Network [70.01091467628068]
  We introduce the Deep Semantic Statistics Matching (D2SM) Denoising Network. It exploits semantic features of pretrained classification networks, then it implicitly matches the probabilistic distribution of clear images at the semantic feature space. By learning to preserve the semantic distribution of denoised images, we empirically find our method significantly improves the denoising capabilities of networks.
  arXiv  Detail & Related papers  (2022-07-19T14:35:42Z)
• Embedding contrastive unsupervised features to cluster in- and out-of-distribution noise in corrupted image datasets [18.19216557948184]
  Using search engines for web image retrieval is a tempting alternative to manual curation when creating an image dataset. Their main drawback remains the proportion of incorrect (noisy) samples retrieved. We propose a two stage algorithm starting with a detection step where we use unsupervised contrastive feature learning. We find that the alignment and uniformity principles of contrastive learning allow OOD samples to be linearly separated from ID samples on the unit hypersphere.
  arXiv  Detail & Related papers  (2022-07-04T16:51:56Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.<n>Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches.<n>We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Poisson2Sparse: Self-Supervised Poisson Denoising From a Single Image [34.27748767631027]
  We present a novel self-supervised learning method for single-image denoising. We approximate traditional iterative optimization algorithms for image denoising with a recurrent neural network. Our method outperforms the state-of-the-art approaches in terms of PSNR and SSIM.
  arXiv  Detail & Related papers  (2022-06-04T00:08:58Z)
• Self-supervised Contrastive Learning for Cross-domain Hyperspectral Image Representation [26.610588734000316]
  This paper introduces a self-supervised learning framework suitable for hyperspectral images that are inherently challenging to annotate. The proposed framework architecture leverages cross-domain CNN, allowing for learning representations from different hyperspectral images. The experimental results demonstrate the advantage of the proposed self-supervised representation over models trained from scratch or other transfer learning methods.
  arXiv  Detail & Related papers  (2022-02-08T16:16:45Z)
• Noise Distribution Adaptive Self-Supervised Image Denoising using Tweedie Distribution and Score Matching [29.97769511276935]
  We show that Tweedie distributions play key roles in modern deep learning era, leading to a distribution independent self-supervised image denoising formula without clean reference images. Specifically, by combining with the recent Noise2Score self-supervised image denoising approach and the saddle point approximation of Tweedie distribution, we can provide a general closed-form denoising formula. We show that the proposed method can accurately estimate noise models and parameters, and provide the state-of-the-art self-supervised image denoising performance in the benchmark dataset and real-world dataset.
  arXiv  Detail & Related papers  (2021-12-05T04:36:08Z)
• Diffusion-Based Representation Learning [65.55681678004038]
  We augment the denoising score matching framework to enable representation learning without any supervised signal. In contrast, the introduced diffusion-based representation learning relies on a new formulation of the denoising score matching objective. Using the same approach, we propose to learn an infinite-dimensional latent code that achieves improvements of state-of-the-art models on semi-supervised image classification.
  arXiv  Detail & Related papers  (2021-05-29T09:26:02Z)
• Spatially Consistent Representation Learning [12.120041613482558]
  We propose a spatially consistent representation learning algorithm (SCRL) for multi-object and location-specific tasks. We devise a novel self-supervised objective that tries to produce coherent spatial representations of a randomly cropped local region. On various downstream localization tasks with benchmark datasets, the proposed SCRL shows significant performance improvements.
  arXiv  Detail & Related papers  (2021-03-10T15:23:45Z)
• Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images [98.82804259905478]
  We present Neighbor2Neighbor to train an effective image denoising model with only noisy images. In detail, input and target used to train a network are images sub-sampled from the same noisy image. A denoising network is trained on sub-sampled training pairs generated in the first stage, with a proposed regularizer as additional loss for better performance.
  arXiv  Detail & Related papers  (2021-01-08T02:03:25Z)
• Set Based Stochastic Subsampling [85.5331107565578]
  We propose a set-based two-stage end-to-end neural subsampling model that is jointly optimized with an textitarbitrary downstream task network. We show that it outperforms the relevant baselines under low subsampling rates on a variety of tasks including image classification, image reconstruction, function reconstruction and few-shot classification.
  arXiv  Detail & Related papers  (2020-06-25T07:36:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.