From Black-box to Causal-box: Towards Building More Interpretable Models

• URL: http://arxiv.org/abs/2510.21998v1
• Date: Fri, 24 Oct 2025 20:03:18 GMT
• Title: From Black-box to Causal-box: Towards Building More Interpretable Models
• Authors: Inwoo Hwang, Yushu Pan, Elias Bareinboim,
• Abstract summary: We introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models.<n>We derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query.
• Score: 57.23201263629627
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Understanding the predictions made by deep learning models remains a central challenge, especially in high-stakes applications. A promising approach is to equip models with the ability to answer counterfactual questions -- hypothetical ``what if?'' scenarios that go beyond the observed data and provide insight into a model reasoning. In this work, we introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models and observational data. We analyze two common model classes -- blackbox and concept-based predictors -- and show that neither is causally interpretable in general. To address this gap, we develop a framework for building models that are causally interpretable by design. Specifically, we derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query. This leads to a fundamental tradeoff between causal interpretability and predictive accuracy, which we characterize by identifying the unique maximal set of features that yields an interpretable model with maximal predictive expressiveness. Experiments corroborate the theoretical findings.

Related papers

• Canonical Representations of Markovian Structural Causal Models: A Framework for Counterfactual Reasoning [0.7106986689736826]
  Counterfactual reasoning aims at answering contrary-to-fact questions like Would have Alice recovered had she taken aspirin?''<n>We propose an alternative approach to structural causal models to represent counterfactuals compatible with a given causal graphical model.<n>We present a normalization procedure to disentangle the (arbitrary and unfalsifiable) counterfactual choice from the (typically testable) interventional constraints.
  arXiv  Detail & Related papers  (2025-07-22T09:13:02Z)
• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors [61.92704516732144]
  We show that the most robust features for correctness prediction are those that play a distinctive causal role in the model's behavior.<n>We propose two methods that leverage causal mechanisms to predict the correctness of model outputs.
  arXiv  Detail & Related papers  (2025-05-17T00:31:39Z)
• Hard to Explain: On the Computational Hardness of In-Distribution Model Interpretation [0.9558392439655016]
  The ability to interpret Machine Learning (ML) models is becoming increasingly essential. Recent work has demonstrated that it is possible to formally assess interpretability by studying the computational complexity of explaining the decisions of various models.
  arXiv  Detail & Related papers  (2024-08-07T17:20:52Z)
• Towards Compositional Interpretability for XAI [3.3768167170511587]
  We present an approach to defining AI models and their interpretability based on category theory. We compare a wide range of AI models as compositional models. We find that what makes the standard 'intrinsically interpretable' models so transparent is brought out most clearly diagrammatically.
  arXiv  Detail & Related papers  (2024-06-25T14:27:03Z)
• Learning Discrete Concepts in Latent Hierarchical Models [73.01229236386148]
  Learning concepts from natural high-dimensional data holds potential in building human-aligned and interpretable machine learning models.<n>We formalize concepts as discrete latent causal variables that are related via a hierarchical causal model.<n>We substantiate our theoretical claims with synthetic data experiments.
  arXiv  Detail & Related papers  (2024-06-01T18:01:03Z)
• Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability [29.459228981179674]
  Post hoc explanations incorrectly attribute high importance to features that are unimportant or non-discriminative for the underlying task. Inherently interpretable models, on the other hand, circumvent these issues by explicitly encoding explanations into model architecture. We propose Distractor Erasure Tuning (DiET), a method that adapts black-box models to be robust to distractor erasure.
  arXiv  Detail & Related papers  (2023-07-27T17:06:02Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Pathologies of Pre-trained Language Models in Few-shot Fine-tuning [50.3686606679048]
  We show that pre-trained language models with few examples show strong prediction bias across labels. Although few-shot fine-tuning can mitigate the prediction bias, our analysis shows models gain performance improvement by capturing non-task-related features. These observations alert that pursuing model performance with fewer examples may incur pathological prediction behavior.
  arXiv  Detail & Related papers  (2022-04-17T15:55:18Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution [3.3758186776249928]
  We aim to address this problem in settings where the predictive model is a black box. We reduce the problem of interpreting a black box predictive model to that of estimating the causal effects of each of the model inputs on the model output. We show how the resulting causal attribution of responsibility for model output to the different model inputs can be used to interpret the predictive model and to explain its predictions.
  arXiv  Detail & Related papers  (2020-08-01T23:20:57Z)
• Explainable Deep Modeling of Tabular Data using TableGraphNet [1.376408511310322]
  We propose a new architecture that produces explainable predictions in the form of additive feature attributions. We show that our explainable model attains the same level of performance as black box models.
  arXiv  Detail & Related papers  (2020-02-12T20:02:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.