Diffusion Model as Representation Learner

• URL: http://arxiv.org/abs/2308.10916v1
• Date: Mon, 21 Aug 2023 00:38:39 GMT
• Title: Diffusion Model as Representation Learner
• Authors: Xingyi Yang and Xinchao Wang
• Abstract summary: Diffusion Probabilistic Models (DPMs) have recently demonstrated impressive results on various generative tasks. We propose a novel knowledge transfer method that leverages the knowledge acquired by DPMs for recognition tasks.
• Score: 86.09969334071478
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Probabilistic Models (DPMs) have recently demonstrated impressive results on various generative tasks.Despite its promises, the learned representations of pre-trained DPMs, however, have not been fully understood. In this paper, we conduct an in-depth investigation of the representation power of DPMs, and propose a novel knowledge transfer method that leverages the knowledge acquired by generative DPMs for recognition tasks. Our study begins by examining the feature space of DPMs, revealing that DPMs are inherently denoising autoencoders that balance the representation learning with regularizing model capacity. To this end, we introduce a novel knowledge transfer paradigm named RepFusion. Our paradigm extracts representations at different time steps from off-the-shelf DPMs and dynamically employs them as supervision for student networks, in which the optimal time is determined through reinforcement learning. We evaluate our approach on several image classification, semantic segmentation, and landmark detection benchmarks, and demonstrate that it outperforms state-of-the-art methods. Our results uncover the potential of DPMs as a powerful tool for representation learning and provide insights into the usefulness of generative models beyond sample generation. The code is available at \url{https://github.com/Adamdad/Repfusion}.

Related papers

• Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax [73.03684002513218]
  We enhance Deep InfoMax (DIM) to enable automatic matching of learned representations to a selected prior distribution. We show that such modification allows for learning uniformly and normally distributed representations. The results indicate a moderate trade-off between the performance on the downstream tasks and quality of DM.
  arXiv  Detail & Related papers  (2024-10-09T15:40:04Z)
• Towards a Theoretical Understanding of Memorization in Diffusion Models [76.85077961718875]
  Diffusion probabilistic models (DPMs) are being employed as mainstream models for Generative Artificial Intelligence (GenAI) We provide a theoretical understanding of memorization in both conditional and unconditional DPMs under the assumption of model convergence. We propose a novel data extraction method named textbfSurrogate condItional Data Extraction (SIDE) that leverages a time-dependent classifier trained on the generated data as a surrogate condition to extract training data from unconditional DPMs.
  arXiv  Detail & Related papers  (2024-10-03T13:17:06Z)
• Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition [58.41784639847413]
  Human emotion is a complex phenomenon conveyed and perceived through facial expressions, vocal tones, body language, and physiological signals. In this paper, a multi-teacher PKD (MT-PKDOT) method with self-distillation is introduced to align diverse teacher representations before distilling them to the student. Results indicate that our proposed method can outperform SOTA PKD methods.
  arXiv  Detail & Related papers  (2024-08-16T22:11:01Z)
• Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles [95.49699178874683]
  We propose DiffDiv, an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs) We show that DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features. We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals.
  arXiv  Detail & Related papers  (2023-11-23T15:47:33Z)
• Efficient Transfer Learning in Diffusion Models via Adversarial Noise [21.609168219488982]
  Diffusion Probabilistic Models (DPMs) have demonstrated substantial promise in image generation tasks. Previous works, like GANs, have tackled the limited data problem by transferring pre-trained models learned with sufficient data. We propose a novel DPMs-based transfer learning method, TAN, to address the limited data problem.
  arXiv  Detail & Related papers  (2023-08-23T06:44:44Z)
• DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models [42.58375679841317]
  We propose a new task, disentanglement of Diffusion Probabilistic Models (DPMs) The task is to automatically discover the inherent factors behind the observations and disentangle the gradient fields of DPM into sub-gradient fields. We devise an unsupervised approach named DisDiff, achieving disentangled representation learning in the framework of DPMs.
  arXiv  Detail & Related papers  (2023-01-31T15:58:32Z)
• Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models [83.75414370493289]
  Diffusion Probabilistic Models (DPMs) have shown a powerful capacity of generating high-quality image samples. Diff-AE have been proposed to explore DPMs for representation learning via autoencoding. We propose textbfPre-trained textbfAutotextbfEncoding (textbfPDAE) to adapt existing pre-trained DPMs to the decoders for image reconstruction.
  arXiv  Detail & Related papers  (2022-12-26T02:37:38Z)
• DDPM-CD: Denoising Diffusion Probabilistic Models as Feature Extractors for Change Detection [31.125812018296127]
  We introduce a novel approach for change detection by pre-training a Deno Diffusionising Probabilistic Model (DDPM) DDPM learns the training data distribution by gradually converting training images into a Gaussian distribution using a Markov chain. During inference (i.e., sampling), they can generate a diverse set of samples closer to the training distribution. Experiments conducted on the LEVIR-CD, WHU-CD, DSIFN-CD, and CDD datasets demonstrate that the proposed DDPM-CD method significantly outperforms the existing change detection methods in terms of F1 score, I
  arXiv  Detail & Related papers  (2022-06-23T17:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.