Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanations

• URL: http://arxiv.org/abs/2506.19630v1
• Date: Tue, 24 Jun 2025 13:54:12 GMT
• Title: Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanations
• Authors: Thomas Decker, Volker Tresp, Florian Buettner,
• Abstract summary: We show that models frequently produce unreliable probability estimates when subjected to explainability-specific perturbations.<n>We introduce ReCalX, a novel approach to recalibrate models for improved perturbation-based explanations.
• Score: 30.47728009839025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Perturbation-based explanations are widely utilized to enhance the transparency of modern machine-learning models. However, their reliability is often compromised by the unknown model behavior under the specific perturbations used. This paper investigates the relationship between uncertainty calibration - the alignment of model confidence with actual accuracy - and perturbation-based explanations. We show that models frequently produce unreliable probability estimates when subjected to explainability-specific perturbations and theoretically prove that this directly undermines explanation quality. To address this, we introduce ReCalX, a novel approach to recalibrate models for improved perturbation-based explanations while preserving their original predictions. Experiments on popular computer vision models demonstrate that our calibration strategy produces explanations that are more aligned with human perception and actual object locations.

Related papers

• Investigating the Impact of Model Instability on Explanations and Uncertainty [43.254616360807496]
  We simulate uncertainty in text input by introducing noise at inference time. We find that high uncertainty doesn't necessarily imply low explanation plausibility. This suggests that noise-augmented models may be better at identifying salient tokens when uncertain.
  arXiv  Detail & Related papers  (2024-02-20T13:41:21Z)
• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• Calibration-Aware Bayesian Learning [37.82259435084825]
  This paper proposes an integrated framework, referred to as calibration-aware Bayesian neural networks (CA-BNNs) It applies both data-dependent or data-independent regularizers while optimizing over a variational distribution as in Bayesian learning. Numerical results validate the advantages of the proposed approach in terms of expected calibration error (ECE) and reliability diagrams.
  arXiv  Detail & Related papers  (2023-05-12T14:19:15Z)
• Calibrate: Interactive Analysis of Probabilistic Model Output [5.444048397001003]
  We present Calibrate, an interactive reliability diagram that is resistant to drawbacks in traditional approaches. We demonstrate the utility of Calibrate through use cases on both real-world and synthetic data.
  arXiv  Detail & Related papers  (2022-07-27T20:01:27Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)
• Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning [0.0]
  We show how calibration error and its variants are almost always incorrect to use given model uncertainty. We show how this mistake can lead to trust in bad models and mistrust in good models.
  arXiv  Detail & Related papers  (2021-12-02T18:26:30Z)
• Quantifying Model Predictive Uncertainty with Perturbation Theory [21.591460685054546]
  We propose a framework for predictive uncertainty quantification of a neural network. We use perturbation theory from quantum physics to formulate a moment decomposition problem. Our approach provides fast model predictive uncertainty estimates with much greater precision and calibration.
  arXiv  Detail & Related papers  (2021-09-22T17:55:09Z)
• Estimation of Bivariate Structural Causal Models by Variational Gaussian Process Regression Under Likelihoods Parametrised by Normalising Flows [74.85071867225533]
  Causal mechanisms can be described by structural causal models. One major drawback of state-of-the-art artificial intelligence is its lack of explainability.
  arXiv  Detail & Related papers  (2021-09-06T14:52:58Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification [58.03725169462616]
  We show theoretically that over-parametrization is not the only reason for over-confidence. We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting. Perhaps surprisingly, we also show that over-confidence is not always the case.
  arXiv  Detail & Related papers  (2021-02-15T21:38:09Z)
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction [49.254162397086006]
  We study explanations based on visual saliency in an image-based age prediction task. We find that presenting model predictions improves human accuracy. However, explanations of various kinds fail to significantly alter human accuracy or trust in the model.
  arXiv  Detail & Related papers  (2020-07-23T20:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.