Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution

• URL: http://arxiv.org/abs/2603.15821v1
• Date: Mon, 16 Mar 2026 18:55:50 GMT
• Title: Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution
• Abstract summary: We find that models with identical predictive behavior can produce substantially different feature attributions.<n>We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery.<n>Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.

Related papers

• Relational Structural Causal Models [52.818153111470394]
  We develop relational structural causal models, extending structural causal models to settings where objects and their relations vary.<n>We propose relational neural causal models, a provably correct approach that outperforms non-relational baselines on simulated traffic scenes.
  arXiv  Detail & Related papers  (2026-06-12T18:54:06Z)
• Evaluating Local Explainability Metrics for Machine Learning Models on Tabular Data [1.423280626666929]
  An explanation can appear plausible to humans but fail to capture the internal reasoning of a model.<n>This paper studies the trustworthiness of local explainability techniques when applied to complex tabular classification tasks.
  arXiv  Detail & Related papers  (2026-05-26T19:38:10Z)
• From Black-box to Causal-box: Towards Building More Interpretable Models [57.23201263629627]
  We introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models.<n>We derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query.
  arXiv  Detail & Related papers  (2025-10-24T20:03:18Z)
• Canonical Representations of Markovian Structural Causal Models: A Framework for Counterfactual Reasoning [0.7106986689736826]
  Counterfactual reasoning aims at answering contrary-to-fact questions like Would have Alice recovered had she taken aspirin?''<n>We propose an alternative approach to structural causal models to represent counterfactuals compatible with a given causal graphical model.<n>We present a normalization procedure to disentangle the (arbitrary and unfalsifiable) counterfactual choice from the (typically testable) interventional constraints.
  arXiv  Detail & Related papers  (2025-07-22T09:13:02Z)
• Hard to Explain: On the Computational Hardness of In-Distribution Model Interpretation [0.9558392439655016]
  The ability to interpret Machine Learning (ML) models is becoming increasingly essential. Recent work has demonstrated that it is possible to formally assess interpretability by studying the computational complexity of explaining the decisions of various models.
  arXiv  Detail & Related papers  (2024-08-07T17:20:52Z)
• Identifiable Latent Neural Causal Models [82.14087963690561]
  Causal representation learning seeks to uncover latent, high-level causal representations from low-level observed data. We determine the types of distribution shifts that do contribute to the identifiability of causal representations. We translate our findings into a practical algorithm, allowing for the acquisition of reliable latent causal representations.
  arXiv  Detail & Related papers  (2024-03-23T04:13:55Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• Why do classifier accuracies show linear trends under distribution shift? [58.40438263312526]
  accuracies of models on one data distribution are approximately linear functions of the accuracies on another distribution. We assume the probability that two models agree in their predictions is higher than what we can infer from their accuracy levels alone. We show that a linear trend must occur when evaluating models on two distributions unless the size of the distribution shift is large.
  arXiv  Detail & Related papers  (2020-12-31T07:24:30Z)
• Structural Causal Models Are (Solvable by) Credal Networks [70.45873402967297]
  Causal inferences can be obtained by standard algorithms for the updating of credal nets. This contribution should be regarded as a systematic approach to represent structural causal models by credal networks. Experiments show that approximate algorithms for credal networks can immediately be used to do causal inference in real-size problems.
  arXiv  Detail & Related papers  (2020-08-02T11:19:36Z)
• Achieving Equalized Odds by Resampling Sensitive Attributes [13.114114427206678]
  We present a flexible framework for learning predictive models that approximately satisfy the equalized odds notion of fairness. This differentiable functional is used as a penalty driving the model parameters towards equalized odds. We develop a formal hypothesis test to detect whether a prediction rule violates this property, the first such test in the literature.
  arXiv  Detail & Related papers  (2020-06-08T00:18:34Z)
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder [52.139696854386976]
  The framework of variational autoencoder (VAE) is commonly used to disentangle independent factors from observations. We propose a new VAE based framework named CausalVAE, which includes a Causal Layer to transform independent factors into causal endogenous ones. Results show that the causal representations learned by CausalVAE are semantically interpretable, and their causal relationship as a Directed Acyclic Graph (DAG) is identified with good accuracy.
  arXiv  Detail & Related papers  (2020-04-18T20:09:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.