Explaining Image Classifiers with Multiscale Directional Image Representation

• URL: http://arxiv.org/abs/2211.12857v3
• Date: Fri, 28 Apr 2023 12:58:15 GMT
• Title: Explaining Image Classifiers with Multiscale Directional Image Representation
• Authors: Stefan Kolek, Robert Windesheim, Hector Andrade Loarca, Gitta Kutyniok, Ron Levie
• Abstract summary: We present ShearletX, a novel mask explanation method for image classifiers based on the shearlet transform. To support our method, we propose a mathematical definition for explanation artifacts and an information theoretic score to evaluate the quality of mask explanations.
• Score: 4.29434097275489
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image classifiers are known to be difficult to interpret and therefore require explanation methods to understand their decisions. We present ShearletX, a novel mask explanation method for image classifiers based on the shearlet transform -- a multiscale directional image representation. Current mask explanation methods are regularized by smoothness constraints that protect against undesirable fine-grained explanation artifacts. However, the smoothness of a mask limits its ability to separate fine-detail patterns, that are relevant for the classifier, from nearby nuisance patterns, that do not affect the classifier. ShearletX solves this problem by avoiding smoothness regularization all together, replacing it by shearlet sparsity constraints. The resulting explanations consist of a few edges, textures, and smooth parts of the original image, that are the most relevant for the decision of the classifier. To support our method, we propose a mathematical definition for explanation artifacts and an information theoretic score to evaluate the quality of mask explanations. We demonstrate the superiority of ShearletX over previous mask based explanation methods using these new metrics, and present exemplary situations where separating fine-detail patterns allows explaining phenomena that were not explainable before.

Related papers

• From Pixels to Components: Eigenvector Masking for Visual Representation Learning [55.567395509598065]
  Predicting masked from visible parts of an image is a powerful self-supervised approach for visual representation learning. We propose an alternative masking strategy that operates on a suitable transformation of the data rather than on the raw pixels.
  arXiv  Detail & Related papers  (2025-02-10T10:06:46Z)
• MaskInversion: Localized Embeddings via Optimization of Explainability Maps [49.50785637749757]
  MaskInversion generates a context-aware embedding for a query image region specified by a mask at test time. It can be used for a broad range of tasks, including open-vocabulary class retrieval, referring expression comprehension, as well as for localized captioning and image generation.
  arXiv  Detail & Related papers  (2024-07-29T14:21:07Z)
• DXAI: Explaining Classification by Image Decomposition [4.013156524547072]
  We propose a new way to visualize neural network classification through a decomposition-based explainable AI (DXAI) Instead of providing an explanation heatmap, our method yields a decomposition of the image into class-agnostic and class-distinct parts.
  arXiv  Detail & Related papers  (2023-12-30T20:52:20Z)
• Multiple Different Black Box Explanations for Image Classifiers [14.182742896993974]
  We describe an algorithm and a tool, MultiReX, for computing multiple explanations of the output of a black-box image classifier for a given image. Our algorithm uses a principled approach based on causal theory. We show that MultiReX finds multiple explanations on 96% of the images in the ImageNet-mini benchmark, whereas previous work finds multiple explanations only on 11%.
  arXiv  Detail & Related papers  (2023-09-25T17:28:28Z)
• MaskSketch: Unpaired Structure-guided Masked Image Generation [56.88038469743742]
  MaskSketch is an image generation method that allows spatial conditioning of the generation result using a guiding sketch as an extra conditioning signal during sampling. We show that intermediate self-attention maps of a masked generative transformer encode important structural information of the input image. Our results show that MaskSketch achieves high image realism and fidelity to the guiding structure.
  arXiv  Detail & Related papers  (2023-02-10T20:27:02Z)
• What You See is What You Classify: Black Box Attributions [61.998683569022006]
  We train a deep network, the Explainer, to predict attributions for a pre-trained black-box classifier, the Explanandum. Unlike most existing approaches, ours is capable of directly generating very distinct class-specific masks. We show that our attributions are superior to established methods both visually and quantitatively.
  arXiv  Detail & Related papers  (2022-05-23T12:30:04Z)
• Explainers in the Wild: Making Surrogate Explainers Robust to Distortions through Perception [77.34726150561087]
  We propose a methodology to evaluate the effect of distortions in explanations by embedding perceptual distances. We generate explanations for images in the Imagenet-C dataset and demonstrate how using a perceptual distances in the surrogate explainer creates more coherent explanations for the distorted and reference images.
  arXiv  Detail & Related papers  (2021-02-22T12:38:53Z)
• This is not the Texture you are looking for! Introducing Novel Counterfactual Explanations for Non-Experts using Generative Adversarial Learning [59.17685450892182]
  counterfactual explanation systems try to enable a counterfactual reasoning by modifying the input image. We present a novel approach to generate such counterfactual image explanations based on adversarial image-to-image translation techniques. Our results show that our approach leads to significantly better results regarding mental models, explanation satisfaction, trust, emotions, and self-efficacy than two state-of-the art systems.
  arXiv  Detail & Related papers  (2020-12-22T10:08:05Z)
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability [5.387323728379395]
  Saliency maps that identify the most informative regions of an image are valuable for model interpretability. A common approach to creating saliency maps involves generating input masks that mask out portions of an image. We show that a masking model can be trained with as few as 10 examples per class and still generate saliency maps with only a 0.7-point increase in localization error.
  arXiv  Detail & Related papers  (2020-10-19T18:00:36Z)
• Causal Intervention for Weakly-Supervised Semantic Segmentation [122.1846968696862]
  We aim to generate better pixel-level pseudo-masks by using only image-level labels. We propose a structural causal model to analyze the causalities among images, contexts, and class labels. Based on it, we develop a new method: Context Adjustment (CONTA), to remove the confounding bias in image-level classification.
  arXiv  Detail & Related papers  (2020-09-26T09:26:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.