Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations

• URL: http://arxiv.org/abs/2605.16651v1
• Date: Fri, 15 May 2026 21:44:16 GMT
• Title: Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations
• Abstract summary: Explanation mechanisms are increasingly used to support transparency and trust in vision-language models.<n>We show that explanation maps can be systematically manipulated while preserving the model's original prediction.<n>We introduce X-Shift, a novel grey-box attack that perturbs patch-level visual representations to redirect explanation heatmaps.
• Score: 0.8594140167290097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Explanation mechanisms are increasingly used to support transparency and trust in vision-language models (VLMs), particularly in settings where model decisions require human oversight. However, the robustness of these explanations remains insufficiently understood. In this work, we investigate whether explanation heatmaps in VLMs, particularly CLIP-based models, faithfully reflect model reasoning under adversarial conditions. We show that explanation maps can be systematically manipulated while preserving the model's original prediction, revealing a disconnect between predictive behavior and explanation faithfulness. To study this vulnerability, we introduce X-Shift, a novel grey-box attack that perturbs patch-level visual representations to redirect explanation heatmaps toward semantically irrelevant regions without altering the predicted output. Unlike conventional adversarial attacks that aim to induce misclassification, X-Shift specifically targets the integrity of the explanation process itself. The attack operates without modifying model parameters and generalizes across multiple CLIP architectures and explanation methods. We evaluate the proposed approach on ImageNet-1k, MS-COCO, and Flickr30K, demonstrating consistent degradation in explanation alignment under imperceptible perturbations while maintaining prediction stability. Furthermore, standard prediction-oriented adversarial attacks fail to reproduce the same explanation-shifting behavior even under substantially larger perturbation budgets. Our findings highlight a fundamental limitation of current explanation mechanisms in VLMs and raise concerns about their use as reliable indicators of model trustworthiness in high-impact applications.

Related papers

• NVExplain: Explaining Time Series Forecasting with Latent Trajectory Analysis and Structure-Preserving Surrogates [2.0320459617688793]
  Time series forecasting models are widely used in high-stakes settings.<n> existing post-hoc methods often ignore temporal dependence and fail to provide horizon-specific explanations.<n>We propose a model-agnostic explainability framework that explains forecasting predictions by attributing each forecast horizon to temporally relevant historical lags.
  arXiv  Detail & Related papers  (2026-08-25T19:14:12Z)
• All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models [9.173232861966595]
  We argue that existing Explainable AI methods suffer from a persistent accuracy-explainability trade-off.<n>We introduce the Rashomon Explanation paradigm, which builds a set of faithful, prediction-guiding explanations.<n>We show that RashomonLLM significantly outperforms state-of-the-art prediction and XAI baselines on both accuracy and explanation quality.
  arXiv  Detail & Related papers  (2026-07-10T15:16:58Z)
• Decoding the Critique Mechanism in Large Reasoning Models [50.821607345799386]
  Large Reasoning Models (LRMs) exhibit backtracking and self-verification mechanisms that enable them to revise intermediate steps and reach correct solutions.<n>We investigate how current LRMs recover from errors by inserting arithmetic mistakes in their intermediate reasoning steps.<n>We find that despite the error propagating through the chain-of-thought, the model still reaches the correct final answer.
  arXiv  Detail & Related papers  (2026-03-17T10:03:30Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• ATEX-CF: Attack-Informed Counterfactual Explanations for Graph Neural Networks [11.482900418658078]
  ATEX-CF is a framework that unifies adversarial attack techniques with counterfactual explanation generation.<n>Our method efficiently integrates both edge additions and deletions, grounded in theory, to explore impactful counterfactuals.<n> Experiments on synthetic and real-world node classification benchmarks demonstrate that ATEX-CF generates faithful, concise, and plausible explanations.
  arXiv  Detail & Related papers  (2026-02-05T22:36:30Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures [70.48661957773449]
  Emergent Misalignment refers to a failure mode in which fine-tuning large language models on narrowly scoped data induces broadly misaligned behavior.<n>Across multiple domains and model families, we find that fine-tuning models on data exhibiting specific character-level dispositions induces substantially stronger and more transferable misalignment than incorrect-advice fine-tuning.
  arXiv  Detail & Related papers  (2026-01-30T15:28:42Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration [34.62583246144584]
  We show that models produce unreliable probability estimates when subjected to explainability-specific perturbations.<n>We introduce ReCalX, a novel approach to recalibrate models for improved explanations while preserving their original predictions.
  arXiv  Detail & Related papers  (2025-11-13T16:04:11Z)
• eXIAA: eXplainable Injections for Adversarial Attack [3.512208543873998]
  We show a new black-box model-agnostic adversarial attack for post-hoc explainable Artificial Intelligence (XAI)<n>The goal of the attack is to modify the original explanations while being undetected by the human eye and maintain the same predicted class.<n>The low requirements of our method expose a critical vulnerability in current explainability methods, raising concerns about their reliability in safety-critical applications.
  arXiv  Detail & Related papers  (2025-11-13T08:42:24Z)
• Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanations [30.47728009839025]
  We show that models frequently produce unreliable probability estimates when subjected to explainability-specific perturbations.<n>We introduce ReCalX, a novel approach to recalibrate models for improved perturbation-based explanations.
  arXiv  Detail & Related papers  (2025-06-24T13:54:12Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• Towards Transparent and Explainable Attention Models [34.0557018891191]
  We first explain why current attention mechanisms in LSTM based encoders can neither provide a faithful nor a plausible explanation of the model's predictions. We propose a modified LSTM cell with a diversity-driven training objective that ensures that the hidden representations learned at different time steps are diverse. Human evaluations indicate that the attention distributions learned by our model offer a plausible explanation of the model's predictions.
  arXiv  Detail & Related papers  (2020-04-29T14:47:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.