Looking in the mirror: A faithful counterfactual explanation method for interpreting deep image classification models

• URL: http://arxiv.org/abs/2509.16822v1
• Date: Sat, 20 Sep 2025 22:21:20 GMT
• Title: Looking in the mirror: A faithful counterfactual explanation method for interpreting deep image classification models
• Abstract summary: Mirror-CFE generates faithful counterfactual explanations by operating directly in the classifier's feature space.<n>Mirror-CFE achieves superior performance in validity while maintaining input resemblance compared to state-of-the-art explanation methods.
• Score: 6.848183976353123
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Counterfactual explanations (CFE) for deep image classifiers aim to reveal how minimal input changes lead to different model decisions, providing critical insights for model interpretation and improvement. However, existing CFE methods often rely on additional image encoders and generative models to create plausible images, neglecting the classifier's own feature space and decision boundaries. As such, they do not explain the intrinsic feature space and decision boundaries learned by the classifier. To address this limitation, we propose Mirror-CFE, a novel method that generates faithful counterfactual explanations by operating directly in the classifier's feature space, treating decision boundaries as mirrors that ``reflect'' feature representations in the mirror. Mirror-CFE learns a mapping function from feature space to image space while preserving distance relationships, enabling smooth transitions between source images and their counterfactuals. Through extensive experiments on four image datasets, we demonstrate that Mirror-CFE achieves superior performance in validity while maintaining input resemblance compared to state-of-the-art explanation methods. Finally, mirror-CFE provides interpretable visualization of the classifier's decision process by generating step-wise transitions that reveal how features evolve as classification confidence changes.

Related papers

• UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation [65.53694602893042]
  We propose a unified visual conditioning framework that fuses ViT and VAE features before VLM encoding.<n>Experiments on two multi-reference generation benchmarks demonstrate that UniCustom consistently improves subject consistency, instruction following, and compositional fidelity.
  arXiv  Detail & Related papers  (2026-05-12T13:10:05Z)
• Concept-based Visual Counterfactual Explanations with Diffusion Models [6.539386622516073]
  C-VCE builds the classifier directly into the generative model via a concept bottleneck layer.<n>Our model lets users toggle on/off semantic concepts during sampling, then adjusts minimally relevant image regions.<n>On benchmarks such as CelebA, C-VCE matches or improves flip rates while producing counterfactuals that are visually closer to the input.
  arXiv  Detail & Related papers  (2026-05-05T15:32:50Z)
• HAMLET-FFD: Hierarchical Adaptive Multi-modal Learning Embeddings Transformation for Face Forgery Detection [6.060036926093259]
  HAMLET-FFD is a cross-domain generalization framework for face forgery detection.<n>It integrates visual evidence with conceptual cues, emulating expert forensic analysis.<n>By design, HAMLET-FFD freezes all pretrained parameters, serving as an external plugin.
  arXiv  Detail & Related papers  (2025-07-28T15:09:52Z)
• Contrastive Learning with Diffusion Features for Weakly Supervised Medical Image Segmentation [12.530950480385554]
  Conditional diffusion model (CDM) has been used as an alternative for generating segmentation masks.<n>We introduce Contrastive Learning with Diffusion Features (CLDF) to train a pixel decoder to map the diffusion features from a frozen CDM to a low-dimensional embedding space for segmentation.
  arXiv  Detail & Related papers  (2025-06-30T01:43:50Z)
• From Visual Explanations to Counterfactual Explanations with Latent Diffusion [11.433402357922414]
  We propose a new approach to tackle two key challenges in recent prominent works.<n>First, we determine which specific counterfactual features are crucial for distinguishing the "concept" of the target class from the original class.<n>Second, we provide valuable explanations for the non-robust classifier without relying on the support of an adversarially robust model.
  arXiv  Detail & Related papers  (2025-04-12T13:04:00Z)
• Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning [81.02648336552421]
  We propose a Multi-Constraint Consistency Learning approach to facilitate the staged enhancement of the encoder and decoder.<n>Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder.<n> Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-23T03:21:33Z)
• Intuitionistic Fuzzy Cognitive Maps for Interpretable Image Classification [2.130156029408832]
  This paper introduces a novel framework, named Interpretable Intuitionistic Fuzzy Cognitive Maps (I2FCMs)<n>Intuitionistic FCMs offer a mechanism to assess the quality of their output through the estimation of hesitancy.<n>The proposed I2FCM framework can be applied to DL models, including CNN, rendering them interpretable.
  arXiv  Detail & Related papers  (2024-08-07T12:58:39Z)
• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• An Integral Projection-based Semantic Autoencoder for Zero-Shot Learning [0.46644955105516456]
  Zero-shot Learning (ZSL) classification categorizes or predicts classes (labels) that are not included in the training set (unseen classes) Recent works proposed different semantic autoencoder (SAE) models where the encoder embeds a visual feature space into the semantic space and the decoder reconstructs the original visual feature space. We propose an integral projection-based semantic autoencoder (IP-SAE) where an encoder projects a visual feature space vectord with the semantic space into a latent representation space.
  arXiv  Detail & Related papers  (2023-06-26T12:06:20Z)
• Reflection Invariance Learning for Few-shot Semantic Segmentation [53.20466630330429]
  Few-shot semantic segmentation (FSS) aims to segment objects of unseen classes in query images with only a few annotated support images. This paper proposes a fresh few-shot segmentation framework to mine the reflection invariance in a multi-view matching manner. Experiments on both PASCAL-$5textiti$ and COCO-$20textiti$ datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-01T15:14:58Z)
• Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations [4.666056064419346]
  The efficient coding hypothesis proposes that the response properties of sensory systems are adapted to the statistics of their inputs. While elegant, information theoretic properties are notoriously difficult to measure in practical settings or to employ as objective functions in optimization. Here we outline the assumptions that allow manifold capacity to be optimized directly, yielding Maximum Manifold Capacity Representations (MMCR)
  arXiv  Detail & Related papers  (2023-03-06T17:26:30Z)
• Diffusion Visual Counterfactual Explanations [51.077318228247925]
  Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts. In this paper, we overcome this by generating Visual Diffusion Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers.
  arXiv  Detail & Related papers  (2022-10-21T09:35:47Z)
• Sparse Visual Counterfactual Explanations in Image Space [50.768119964318494]
  We present a novel model for visual counterfactual explanations in image space. We show that it can be used to detect undesired behavior of ImageNet classifiers due to spurious features in the ImageNet dataset.
  arXiv  Detail & Related papers  (2022-05-16T20:23:11Z)
• Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation [93.83369981759996]
  We propose a self-supervised equivariant attention mechanism (SEAM) to discover additional supervision and narrow the gap. Our method is based on the observation that equivariance is an implicit constraint in fully supervised semantic segmentation. We propose consistency regularization on predicted CAMs from various transformed images to provide self-supervision for network learning.
  arXiv  Detail & Related papers  (2020-04-09T14:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.