Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set

• URL: http://arxiv.org/abs/2601.08703v1
• Date: Tue, 13 Jan 2026 16:31:11 GMT
• Title: Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set
• Abstract summary: We propose three principles of explanation evaluation and a new method "AXE" to evaluate the quality of feature-importance explanations.<n>We show how evaluation metrics that rely on comparing model explanations against ideal ground truth explanations obscure behavioral differences within a Rashomon set.
• Score: 10.788168368884804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explainable artificial intelligence (XAI) is concerned with producing explanations indicating the inner workings of models. For a Rashomon set of similarly performing models, explanations provide a way of disambiguating the behavior of individual models, helping select models for deployment. However explanations themselves can vary depending on the explainer used, and need to be evaluated. In the paper "Evaluating Model Explanations without Ground Truth", we proposed three principles of explanation evaluation and a new method "AXE" to evaluate the quality of feature-importance explanations. We go on to illustrate how evaluation metrics that rely on comparing model explanations against ideal ground truth explanations obscure behavioral differences within a Rashomon set. Explanation evaluation aligned with our proposed principles would highlight these differences instead, helping select models from the Rashomon set. The selection of alternate models from the Rashomon set can maintain identical predictions but mislead explainers into generating false explanations, and mislead evaluation methods into considering the false explanations to be of high quality. AXE, our proposed explanation evaluation method, can detect this adversarial fairwashing of explanations with a 100% success rate. Unlike prior explanation evaluation strategies such as those based on model sensitivity or ground truth comparison, AXE can determine when protected attributes are used to make predictions.

Related papers

• Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning [4.570003973862485]
  Multiple machine learning models can achieve near-equivalent predictive performance on the same task, yet provide divergent feature-based explanations.<n>This is called the Rashomon effect of (explainable) machine learning, and it raises the question of which explanations, if any, are trustworthy.<n>We propose a framework based on metamorphic testing that assesses explanation faithfulness without requiring ground-truth labels.
  arXiv  Detail & Related papers  (2026-06-04T11:57:26Z)
• Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales [1.8772057593980798]
  This paper compares two families of explanations for question answering models: verbalized feature attributions and self-generated rationales.<n>We analyze how explanation source, verbalization strategy, and feature granularity affect the simulatability of explanations.
  arXiv  Detail & Related papers  (2026-05-31T10:35:35Z)
• Evaluating Model Explanations without Ground Truth [12.35100095333756]
  We propose a ground-truth Agnostic eXplanation Evaluation framework (AXE) for evaluating and comparing model explanations.<n>AXE does not require access to ideal ground-truth explanations for comparison, or rely on model sensitivity - providing an independent measure of explanation quality.
  arXiv  Detail & Related papers  (2025-05-15T15:22:06Z)
• Selective Explanations [14.312717332216073]
  A machine learning model is trained to predict feature attribution scores with only one inference. Despite their efficiency, amortized explainers can produce inaccurate predictions and misleading explanations. We propose selective explanations, a novel feature attribution method that detects when amortized explainers generate low-quality explanations.
  arXiv  Detail & Related papers  (2024-05-29T23:08:31Z)
• On Generating Monolithic and Model Reconciling Explanations in Probabilistic Scenarios [46.24262986854885]
  We propose a novel framework for generating probabilistic monolithic explanations and model reconciling explanations.<n>For monolithic explanations, our approach integrates uncertainty by utilizing probabilistic logic to increase the probability of the explanandum.<n>For model reconciling explanations, we propose a framework that extends the logic-based variant of the model reconciliation problem to account for probabilistic human models.
  arXiv  Detail & Related papers  (2024-05-29T16:07:31Z)
• An Axiomatic Approach to Model-Agnostic Concept Explanations [67.84000759813435]
  We propose an approach to concept explanations that satisfy three natural axioms: linearity, recursivity, and similarity. We then establish connections with previous concept explanation methods, offering insight into their varying semantic meanings.
  arXiv  Detail & Related papers  (2024-01-12T20:53:35Z)
• BELLA: Black box model Explanations by Local Linear Approximations [10.71090921516854]
  BELLA is a deterministic model-agnostic post-hoc approach for explaining the individual predictions of regression black-box models.<n>BELLA maximizes the size of the neighborhood to which the linear model applies so that the explanations are accurate, simple, general, and robust.
  arXiv  Detail & Related papers  (2023-05-18T21:22:23Z)
• Explainability in Process Outcome Prediction: Guidelines to Obtain Interpretable and Faithful Models [77.34726150561087]
  We define explainability through the interpretability of the explanations and the faithfulness of the explainability model in the field of process outcome prediction. This paper contributes a set of guidelines named X-MOP which allows selecting the appropriate model based on the event log specifications.
  arXiv  Detail & Related papers  (2022-03-30T05:59:50Z)
• Evaluating Explanations for Reading Comprehension with Realistic Counterfactuals [26.641834518599303]
  We propose a methodology to evaluate explanations for machine reading comprehension tasks. An explanation should allow us to understand the RC model's high-level behavior with respect to a set of realistic counterfactual input scenarios. Our analysis suggests that pairwise explanation techniques are better suited to RC than token-level attributions.
  arXiv  Detail & Related papers  (2021-04-09T17:55:21Z)
• Contrastive Explanations for Model Interpretability [77.92370750072831]
  We propose a methodology to produce contrastive explanations for classification models. Our method is based on projecting model representation to a latent space. Our findings shed light on the ability of label-contrastive explanations to provide a more accurate and finer-grained interpretability of a model's decision.
  arXiv  Detail & Related papers  (2021-03-02T00:36:45Z)
• Evaluating Explanations: How much do explanations from the teacher aid students? [103.05037537415811]
  We formalize the value of explanations using a student-teacher paradigm that measures the extent to which explanations improve student models in learning. Unlike many prior proposals to evaluate explanations, our approach cannot be easily gamed, enabling principled, scalable, and automatic evaluation of attributions.
  arXiv  Detail & Related papers  (2020-12-01T23:40:21Z)
• The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets [61.66584140190247]
  We show that feature-based explanations pose problems even for explaining trivial models. We show that two popular classes of explainers, Shapley explainers and minimal sufficient subsets explainers, target fundamentally different types of ground-truth explanations.
  arXiv  Detail & Related papers  (2020-09-23T09:45:23Z)
• Evaluations and Methods for Explanation through Robustness Analysis [117.7235152610957]
  We establish a novel set of evaluation criteria for such feature based explanations by analysis. We obtain new explanations that are loosely necessary and sufficient for a prediction. We extend the explanation to extract the set of features that would move the current prediction to a target class.
  arXiv  Detail & Related papers  (2020-05-31T05:52:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.