Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance

• URL: http://arxiv.org/abs/2208.08664v1
• Date: Thu, 18 Aug 2022 06:51:23 GMT
• Title: Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance
• Authors: Bahjat Kawar, Roy Ganz, Michael Elad
• Abstract summary: In order to obtain class-conditional generation, it was suggested to guide the diffusion process by gradients from a time-dependent classifier. While the idea is theoretically sound, deep learning-based classifiers are infamously susceptible to gradient-based adversarial attacks. We utilize this observation by defining and training a time-dependent adversarially robust classifier and use it as guidance for a generative diffusion model.
• Score: 17.929524924008962
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Denoising diffusion probabilistic models (DDPMs) are a recent family of generative models that achieve state-of-the-art results. In order to obtain class-conditional generation, it was suggested to guide the diffusion process by gradients from a time-dependent classifier. While the idea is theoretically sound, deep learning-based classifiers are infamously susceptible to gradient-based adversarial attacks. Therefore, while traditional classifiers may achieve good accuracy scores, their gradients are possibly unreliable and might hinder the improvement of the generation results. Recent work discovered that adversarially robust classifiers exhibit gradients that are aligned with human perception, and these could better guide a generative process towards semantically meaningful images. We utilize this observation by defining and training a time-dependent adversarially robust classifier and use it as guidance for a generative diffusion model. In experiments on the highly challenging and diverse ImageNet dataset, our scheme introduces significantly more intelligible intermediate gradients, better alignment with theoretical findings, as well as improved generation results under several evaluation metrics. Furthermore, we conduct an opinion survey whose findings indicate that human raters prefer our method's results.

Related papers

• Classifier Guidance Enhances Diffusion-based Adversarial Purification by Preserving Predictive Information [75.36597470578724]
  Adversarial purification is one of the promising approaches to defend neural networks against adversarial attacks. We propose gUided Purification (COUP) algorithm, which purifies while keeping away from the classifier decision boundary. Experimental results show that COUP can achieve better adversarial robustness under strong attack methods.
  arXiv  Detail & Related papers  (2024-08-12T02:48:00Z)
• Diffusion Model with Perceptual Loss [4.67483805599143]
  Diffusion models trained with mean squared error loss tend to generate unrealistic samples. We show that the effectiveness of classifier-free guidance partly originates from it being a form of implicit perceptual guidance. We propose a novel self-perceptual objective that results in diffusion models capable of generating more realistic samples.
  arXiv  Detail & Related papers  (2023-12-30T01:24:25Z)
• DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentation [48.25619775814776]
  This paper proposes DiffAug, a novel unsupervised contrastive learning technique with diffusion mode-based positive data generation. DiffAug consists of a semantic encoder and a conditional diffusion model; the conditional diffusion model generates new positive samples conditioned on the semantic encoding. Experimental evaluations show that DiffAug outperforms hand-designed and SOTA model-based augmentation methods on DNA sequence, visual, and bio-feature datasets.
  arXiv  Detail & Related papers  (2023-09-10T13:28:46Z)
• Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling [69.60713300418467]
  Learning to jump is a general recipe for generative modeling of various types of data. We demonstrate when learning to jump is expected to perform comparably to learning to denoise, and when it is expected to perform better.
  arXiv  Detail & Related papers  (2023-05-28T05:38:28Z)
• Exploring Compositional Visual Generation with Latent Classifier Guidance [19.48538300223431]
  We train latent diffusion models and auxiliary latent classifiers to facilitate non-linear navigation of latent representation generation. We show that such conditional generation achieved by latent classifier guidance provably maximizes a lower bound of the conditional log probability during training. We show that this paradigm based on latent classifier guidance is agnostic to pre-trained generative models, and present competitive results for both image generation and sequential manipulation of real and synthetic images.
  arXiv  Detail & Related papers  (2023-04-25T03:02:58Z)
• End-to-End Diffusion Latent Optimization Improves Classifier Guidance [81.27364542975235]
  Direct Optimization of Diffusion Latents (DOODL) is a novel guidance method. It enables plug-and-play guidance by optimizing diffusion latents. It outperforms one-step classifier guidance on computational and human evaluation metrics.
  arXiv  Detail & Related papers  (2023-03-23T22:43:52Z)
• Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise [62.997667081978825]
  In high-risk environments, deep learning models need to be able to judge their uncertainty and reject inputs when there is a significant chance of misclassification. We conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise.
  arXiv  Detail & Related papers  (2023-01-03T11:34:36Z)
• Improving Adversarial Robustness by Contrastive Guided Diffusion Process [19.972628281993487]
  We propose Contrastive-Guided Diffusion Process (Contrastive-DP) to guide the diffusion model in data generation. We show that enhancing the distinguishability among the generated data is critical for improving adversarial robustness.
  arXiv  Detail & Related papers  (2022-10-18T07:20:53Z)
• Deblurring via Stochastic Refinement [85.42730934561101]
  We present an alternative framework for blind deblurring based on conditional diffusion models. Our method is competitive in terms of distortion metrics such as PSNR.
  arXiv  Detail & Related papers  (2021-12-05T04:36:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.