On the Faithfulness of Vision Transformer Explanations

• URL: http://arxiv.org/abs/2404.01415v2
• Date: Sat, 16 Nov 2024 04:37:39 GMT
• Title: On the Faithfulness of Vision Transformer Explanations
• Abstract summary: Post-hoc explanations assign salience scores to input pixels, providing human-understandable heatmaps. We introduce Salience-guided Faithfulness Coefficient (SaCo), a novel evaluation metric leveraging essential information of salience distribution. Our SaCo demonstrates that the use of gradient and multi-layer aggregation can markedly enhance the faithfulness of attention-based explanation.
• Score: 23.963269554498716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To interpret Vision Transformers, post-hoc explanations assign salience scores to input pixels, providing human-understandable heatmaps. However, whether these interpretations reflect true rationales behind the model's output is still underexplored. To address this gap, we study the faithfulness criterion of explanations: the assigned salience scores should represent the influence of the corresponding input pixels on the model's predictions. To evaluate faithfulness, we introduce Salience-guided Faithfulness Coefficient (SaCo), a novel evaluation metric leveraging essential information of salience distribution. Specifically, we conduct pair-wise comparisons among distinct pixel groups and then aggregate the differences in their salience scores, resulting in a coefficient that indicates the explanation's degree of faithfulness. Our explorations reveal that current metrics struggle to differentiate between advanced explanation methods and Random Attribution, thereby failing to capture the faithfulness property. In contrast, our proposed SaCo offers a reliable faithfulness measurement, establishing a robust metric for interpretations. Furthermore, our SaCo demonstrates that the use of gradient and multi-layer aggregation can markedly enhance the faithfulness of attention-based explanation, shedding light on potential paths for advancing Vision Transformer explainability.

Related papers

• Zero-Shot Faithful Textual Explanations via Directional-Derivative Influence on Predictions [12.961180148172199]
  textual explanations aim to make image classifiers more transparent by probing their internal representations.<n>FaithTrace measures how much the representation induced by the explanation changes the class logit.<n>We extend this influence score into quantitative evaluation metrics, helping fill the gap in faithfulness evaluation for textual explanations.
  arXiv  Detail & Related papers  (2026-05-16T08:39:05Z)
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs [54.16324124242172]
  Continuous latent-space reasoning offers a compact alternative to textual chain-of-thought for multimodal models.<n>We identify a previously overlooked optimization pathology in existing latent visual reasoning methods.<n>We show that inference-time latent optimization, without any parameter updates, effectively unleashes the suppressed reasoning capacity of visual latents.
  arXiv  Detail & Related papers  (2026-05-04T15:36:12Z)
• Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance [57.17102098930037]
  Large language models (LLMs) achieve strong performance and have revolutionized NLP.<n>Their lack of explainability keeps them treated as black boxes, limiting their use in domains that demand transparency and trust.<n>We introduce a training-free method that enhances faithfulness by guiding explanation generation through attention-level interventions.
  arXiv  Detail & Related papers  (2026-04-15T18:32:32Z)
• Do Sparse Subnetworks Exhibit Cognitively Aligned Attention? Effects of Pruning on Saliency Map Fidelity, Sparsity, and Concept Coherence [2.7415651415305597]
  We investigate how magnitude-based pruning followed by fine-tuning affects both low-level saliency maps and high-level concept representations.<n>Our results show that light-to-moderate pruning improves saliency-map focus and faithfulness while retaining distinct, semantically meaningful concepts.<n>In contrast, aggressive pruning merges heterogeneous features, reducing saliency map sparsity and concept coherence despite maintaining accuracy.
  arXiv  Detail & Related papers  (2025-09-23T20:10:23Z)
• I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? [76.15163242945813]
  Large language models (LLMs) have led many to conclude that they exhibit a form of intelligence.<n>We introduce a novel generative model that generates tokens on the basis of human-interpretable concepts represented as latent discrete variables.
  arXiv  Detail & Related papers  (2025-03-12T01:21:17Z)
• Uncovering Uncertainty in Transformer Inference [1.1726838461147573]
  We explore the Iterative Inference Hypothesis (IIH) within the context of transformer-based language models. Our findings provide empirical support for the IIH, showing that the nth token embedding in the residual stream follows a trajectory of decreasing loss. We introduce a method utilizing cross-entropy to detect this uncertainty and demonstrate its potential to distinguish between correct and incorrect token generations on a dataset of idioms.
  arXiv  Detail & Related papers  (2024-12-08T00:46:10Z)
• Rethinking Distance Metrics for Counterfactual Explainability [53.436414009687]
  We investigate a framing for counterfactual generation methods that considers counterfactuals not as independent draws from a region around the reference, but as jointly sampled with the reference from the underlying data distribution. We derive a distance metric, tailored for counterfactual similarity that can be applied to a broad range of settings.
  arXiv  Detail & Related papers  (2024-10-18T15:06:50Z)
• Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models [29.67884478799914]
  Large Language Models (LLMs) are capable of generating persuasive Natural Language Explanations (NLEs) to justify their answers. Recent studies have proposed various methods to measure the faithfulness of NLEs, typically by inserting perturbations at the explanation or feature level. We argue that these approaches are neither comprehensive nor correctly designed according to the established definition of faithfulness.
  arXiv  Detail & Related papers  (2024-10-18T03:45:42Z)
• Goodhart's Law Applies to NLP's Explanation Benchmarks [57.26445915212884]
  We critically examine two sets of metrics: the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics. We show that we can inflate a model's comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution test inputs. Our results raise doubts about the ability of current metrics to guide explainability research, underscoring the need for a broader reassessment of what precisely these metrics are intended to capture.
  arXiv  Detail & Related papers  (2023-08-28T03:03:03Z)
• Beyond Model Interpretability: On the Faithfulness and Adversarial Robustness of Contrastive Textual Explanations [2.543865489517869]
  This work motivates textual counterfactuals by laying the ground for a novel evaluation scheme inspired by the faithfulness of explanations. Experiments on sentiment analysis data show that the connectedness of counterfactuals to their original counterparts is not obvious in both models.
  arXiv  Detail & Related papers  (2022-10-17T09:50:02Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Where and What? Examining Interpretable Disentangled Representations [96.32813624341833]
  Capturing interpretable variations has long been one of the goals in disentanglement learning. Unlike the independence assumption, interpretability has rarely been exploited to encourage disentanglement in the unsupervised setting. In this paper, we examine the interpretability of disentangled representations by investigating two questions: where to be interpreted and what to be interpreted.
  arXiv  Detail & Related papers  (2021-04-07T11:22:02Z)
• Explainers in the Wild: Making Surrogate Explainers Robust to Distortions through Perception [77.34726150561087]
  We propose a methodology to evaluate the effect of distortions in explanations by embedding perceptual distances. We generate explanations for images in the Imagenet-C dataset and demonstrate how using a perceptual distances in the surrogate explainer creates more coherent explanations for the distorted and reference images.
  arXiv  Detail & Related papers  (2021-02-22T12:38:53Z)
• Disentangling Action Sequences: Discovering Correlated Samples [6.179793031975444]
  We demonstrate the data itself plays a crucial role in disentanglement and instead of the factors, and the disentangled representations align the latent variables with the action sequences. We propose a novel framework, fractional variational autoencoder (FVAE) to disentangle the action sequences with different significance step-by-step. Experimental results on dSprites and 3D Chairs show that FVAE improves the stability of disentanglement.
  arXiv  Detail & Related papers  (2020-10-17T07:37:50Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Assessing the Reliability of Visual Explanations of Deep Models with Adversarial Perturbations [15.067369314723958]
  We propose an objective measure to evaluate the reliability of explanations of deep models. Our approach is based on changes in the network's outcome resulting from the perturbation of input images in an adversarial way. We also propose a straightforward application of our approach to clean relevance maps, creating more interpretable maps without any loss in essential explanation.
  arXiv  Detail & Related papers  (2020-04-22T19:57:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.