Diffusion Visual Counterfactual Explanations

• URL: http://arxiv.org/abs/2210.11841v1
• Date: Fri, 21 Oct 2022 09:35:47 GMT
• Title: Diffusion Visual Counterfactual Explanations
• Authors: Maximilian Augustin, Valentyn Boreiko, Francesco Croce, Matthias Hein
• Abstract summary: Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts. In this paper, we overcome this by generating Visual Diffusion Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers.
• Score: 51.077318228247925
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image classifier. They are 'small' but 'realistic' semantic changes of the image changing the classifier decision. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts, or are limited to image classification problems with few classes. In this paper, we overcome this by generating Diffusion Visual Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers via a diffusion process. Two modifications to the diffusion process are key for our DVCEs: first, an adaptive parameterization, whose hyperparameters generalize across images and models, together with distance regularization and late start of the diffusion process, allow us to generate images with minimal semantic changes to the original ones but different classification. Second, our cone regularization via an adversarially robust model ensures that the diffusion process does not converge to trivial non-semantic changes, but instead produces realistic images of the target class which achieve high confidence by the classifier.

Related papers

• Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning [71.14084801851381]
  Change captioning aims to succinctly describe the semantic change between a pair of similar images. Most existing methods directly capture the difference between them, which risk obtaining error-prone difference features. We propose a distractors-immune representation learning network that correlates the corresponding channels of two image representations.
  arXiv  Detail & Related papers  (2024-07-16T13:00:33Z)
• Semantic Guidance Tuning for Text-To-Image Diffusion Models [3.3881449308956726]
  We propose a training-free approach that modulates the guidance direction of diffusion models during inference. We first decompose the prompt semantics into a set of concepts, and monitor the guidance trajectory in relation to each concept. Based on this observation, we devise a technique to steer the guidance direction towards any concept from which the model diverges.
  arXiv  Detail & Related papers  (2023-12-26T09:02:17Z)
• Effective Data Augmentation With Diffusion Models [65.09758931804478]
  We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained text-to-image diffusion models. Our method edits images to change their semantics using an off-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples. We evaluate our approach on few-shot image classification tasks, and on a real-world weed recognition task, and observe an improvement in accuracy in tested domains.
  arXiv  Detail & Related papers  (2023-02-07T20:42:28Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• Semantic-Conditional Diffusion Networks for Image Captioning [116.86677915812508]
  We propose a new diffusion model based paradigm tailored for image captioning, namely Semantic-Conditional Diffusion Networks (SCD-Net) In SCD-Net, multiple Diffusion Transformer structures are stacked to progressively strengthen the output sentence with better visional-language alignment and linguistical coherence. Experiments on COCO dataset demonstrate the promising potential of using diffusion models in the challenging image captioning task.
  arXiv  Detail & Related papers  (2022-12-06T16:08:16Z)
• Encoding Robustness to Image Style via Adversarial Feature Perturbations [72.81911076841408]
  We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce robust models. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training.
  arXiv  Detail & Related papers  (2020-09-18T17:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.