Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

• URL: http://arxiv.org/abs/2606.29346v1
• Date: Sun, 28 Jun 2026 11:29:15 GMT
• Title: Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models
• Abstract summary: Post-hoc explanation methods are routinely used to interpret scientific machine learning models.<n>We argue that the transition may be taken to be secured once the model is reliable enough and the explanation faithful enough.<n> Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works.
• Score: 0.6875312133832079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.

Related papers

• Relational Structural Causal Models [52.818153111470394]
  We develop relational structural causal models, extending structural causal models to settings where objects and their relations vary.<n>We propose relational neural causal models, a provably correct approach that outperforms non-relational baselines on simulated traffic scenes.
  arXiv  Detail & Related papers  (2026-06-12T18:54:06Z)
• From Black-box to Causal-box: Towards Building More Interpretable Models [57.23201263629627]
  We introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models.<n>We derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query.
  arXiv  Detail & Related papers  (2025-10-24T20:03:18Z)
• When Counterfactual Reasoning Fails: Chaos and Real-World Complexity [1.9223856107206057]
  We investigate the limitations of counterfactual reasoning within the framework of Structural Causal Models.<n>We find that realistic assumptions, such as low degrees of model uncertainty or chaotic dynamics, can result in counterintuitive outcomes.<n>This work urges caution when applying counterfactual reasoning in settings characterized by chaos and uncertainty.
  arXiv  Detail & Related papers  (2025-03-31T08:14:51Z)
• Causally Reliable Concept Bottleneck Models [4.411356026951205]
  Concept-based models fail to account for the true causal mechanisms underlying the target phenomena represented in the data.<n>We propose Causally reliable Concept Bottleneck Models (C$2$BMs), a class of concept-based architectures that enforce reasoning through a bottleneck of concepts structured according to a model of the real-world causal mechanisms.<n>We show that C$2$BMs are more interpretable, causally reliable, and improve responsiveness to interventions w.r.t. standard opaque and concept-based models, while maintaining their accuracy.
  arXiv  Detail & Related papers  (2025-03-06T12:06:54Z)
• Interpretability Needs a New Paradigm [49.134097841837715]
  Interpretability is the study of explaining models in understandable terms to humans. At the core of this debate is how each paradigm ensures its explanations are faithful, i.e., true to the model's behavior. This paper's position is that we should think about new paradigms while staying vigilant regarding faithfulness.
  arXiv  Detail & Related papers  (2024-05-08T19:31:06Z)
• Faithful Model Explanations through Energy-Constrained Conformal Counterfactuals [16.67633872254042]
  Counterfactual explanations offer an intuitive and straightforward way to explain black-box models. Existing work has primarily relied on surrogate models to learn how the input data is distributed. We propose a novel algorithmic framework for generating Energy-Constrained Conformal Counterfactuals that are only as plausible as the model permits.
  arXiv  Detail & Related papers  (2023-12-17T08:24:44Z)
• Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability [29.459228981179674]
  Post hoc explanations incorrectly attribute high importance to features that are unimportant or non-discriminative for the underlying task. Inherently interpretable models, on the other hand, circumvent these issues by explicitly encoding explanations into model architecture. We propose Distractor Erasure Tuning (DiET), a method that adapts black-box models to be robust to distractor erasure.
  arXiv  Detail & Related papers  (2023-07-27T17:06:02Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Uncertain Evidence in Probabilistic Models and Stochastic Simulators [80.40110074847527]
  We consider the problem of performing Bayesian inference in probabilistic models where observations are accompanied by uncertainty, referred to as uncertain evidence' We explore how to interpret uncertain evidence, and by extension the importance of proper interpretation as it pertains to inference about latent variables. We devise concrete guidelines on how to account for uncertain evidence and we provide new insights, particularly regarding consistency.
  arXiv  Detail & Related papers  (2022-10-21T20:32:59Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Transport-based Counterfactual Models [0.0]
  State-of-the-art models to compute counterfactuals are either unrealistic or unfeasible. We address the problem of designing realistic and feasible counterfactuals in the absence of a causal model. We argue that optimal transport theory defines relevant transport-based counterfactual models.
  arXiv  Detail & Related papers  (2021-08-30T07:28:19Z)
• Structural Causal Models Are (Solvable by) Credal Networks [70.45873402967297]
  Causal inferences can be obtained by standard algorithms for the updating of credal nets. This contribution should be regarded as a systematic approach to represent structural causal models by credal networks. Experiments show that approximate algorithms for credal networks can immediately be used to do causal inference in real-size problems.
  arXiv  Detail & Related papers  (2020-08-02T11:19:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.