All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models

• URL: http://arxiv.org/abs/2607.09502v1
• Date: Fri, 10 Jul 2026 15:16:58 GMT
• Title: All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models
• Abstract summary: We argue that existing Explainable AI methods suffer from a persistent accuracy-explainability trade-off.<n>We introduce the Rashomon Explanation paradigm, which builds a set of faithful, prediction-guiding explanations.<n>We show that RashomonLLM significantly outperforms state-of-the-art prediction and XAI baselines on both accuracy and explanation quality.
• Score: 9.173232861966595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explaining machine-learning models is increasingly important for decision-making and consumer trust, yet it is widely believed to come at a cost: existing Explainable AI (XAI) methods suffer from a persistent accuracy-explainability trade-off. We argue that this trade-off is not fundamental, but an artifact of treating explanation and prediction as separate objectives; when properly coupled, they become complementary, so that equipping a model to explain itself improves, rather than degrades, its accuracy. We introduce the Rashomon Explanation paradigm, which builds a set of faithful, prediction-guiding explanations rather than a single one, and prove that this set is generally non-empty and that explanation fidelity bounds the performance of the models it guides. To explore this set, we propose RashomonLLM, an Explanation-Prediction-Reflection agentic workflow that generates explanations in natural language by iteratively aligning them with predictions, and we prove it converges and recovers the full set. Across customer-churn classification, clinical survival regression, and industrial click-through prediction on large-scale live-streaming logs, RashomonLLM significantly outperforms state-of-the-art prediction and XAI baselines on both accuracy and explanation quality, with gains driven by explanation fidelity and robust to distribution shifts, temporal splits, and seeds. Our framework thus advances business performance while laying the groundwork for consumer trust.

Related papers

• XAI-Refine: An Automated Explanation-Knowledge Loop for Brain-Age Prediction [8.369324356989523]
  We propose an automated explanation loop for brain-age prediction from resting-state functional connectivity.<n>XAI-Refine consolidates complementary post-hoc analyses across repeated training runs into reliable, structured model explanations.<n>It converts each reliable explanation into a neutral neurobiological question, retrieves and verifies relevant literature, and compiles the verified evidence into an admissible set.
  arXiv  Detail & Related papers  (2026-09-08T19:40:42Z)
• Towards A Unified Information Bottleneck Framework for Time Series Explanations [27.646760316510356]
  We show that existing explainers are vulnerable to trivial solutions and distributional shifts.<n>We propose a unified objective function for explainable time series learning that bridges attribution and counterfactual reasoning.<n>Our formulation explicitly prevents trivial explanations and out-of-distribution counterfactuals.
  arXiv  Detail & Related papers  (2026-08-26T15:14:52Z)
• Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations [0.8594140167290097]
  Explanation mechanisms are increasingly used to support transparency and trust in vision-language models.<n>We show that explanation maps can be systematically manipulated while preserving the model's original prediction.<n>We introduce X-Shift, a novel grey-box attack that perturbs patch-level visual representations to redirect explanation heatmaps.
  arXiv  Detail & Related papers  (2026-05-15T21:44:16Z)
• Pando: Do Interpretability Methods Work When Models Won't Explain Themselves? [53.07826484214082]
  We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis.<n>Pando predicts held-out model decisions from $10$ labeled query-response pairs.<n>When explanations are faithful, black-box elicitation matches or exceeds all white-box methods.
  arXiv  Detail & Related papers  (2026-04-13T06:42:24Z)
• Faithful and Interpretable Explanations for Complex Ensemble Time Series Forecasts using Surrogate Models and Forecastability Analysis [1.5751034894694789]
  We develop a surrogate-based explanation methodology that bridges the accuracy-interpretability gap.<n>We integrate spectral predictability analysis to quantify each series' inherent forecastability.<n>The resulting framework delivers interpretable, instance-level explanations for state-of-the-art ensemble forecasts.
  arXiv  Detail & Related papers  (2025-10-09T18:49:45Z)
• I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? [76.15163242945813]
  Large language models (LLMs) have led many to conclude that they exhibit a form of intelligence.<n>We introduce a novel generative model that generates tokens on the basis of human-interpretable concepts represented as latent discrete variables.
  arXiv  Detail & Related papers  (2025-03-12T01:21:17Z)
• Explaining Language Models' Predictions with High-Impact Concepts [11.47612457613113]
  We propose a complete framework for extending concept-based interpretability methods to NLP. We optimize for features whose existence causes the output predictions to change substantially. Our method achieves superior results on predictive impact, usability, and faithfulness compared to the baselines.
  arXiv  Detail & Related papers  (2023-05-03T14:48:27Z)
• VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives [84.48039784446166]
  We show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason metrics. Our best performing method, Visual Feature Importance Supervision (VisFIS), outperforms strong baselines on benchmark VQA datasets. Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful.
  arXiv  Detail & Related papers  (2022-06-22T17:02:01Z)
• Counterfactual Explanations for Predictive Business Process Monitoring [0.90238471756546]
  We propose LORELEY, a counterfactual explanation technique for predictive process monitoring. LORELEY can approximate prediction models with an average fidelity of 97.69% and generate realistic counterfactual explanations.
  arXiv  Detail & Related papers  (2022-02-24T11:01:20Z)
• Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set [50.67431815647126]
  Post-hoc global/local feature attribution methods are being progressively employed to understand machine learning models. We show that partial orders of local/global feature importance arise from this methodology. We show that every relation among features present in these partial orders also holds in the rankings provided by existing approaches.
  arXiv  Detail & Related papers  (2021-10-26T02:53:14Z)
• Do not explain without context: addressing the blind spot of model explanations [2.280298858971133]
  This paper highlights a blind spot which is often overlooked when monitoring and auditing machine learning models. We discuss that many model explanations depend directly or indirectly on the choice of the referenced data distribution. We showcase examples where small changes in the distribution lead to drastic changes in the explanations, such as a change in trend or, alarmingly, a conclusion.
  arXiv  Detail & Related papers  (2021-05-28T12:48:40Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)
• Towards Interpretable Natural Language Understanding with Explanations as Latent Variables [146.83882632854485]
  We develop a framework for interpretable natural language understanding that requires only a small set of human annotated explanations for training. Our framework treats natural language explanations as latent variables that model the underlying reasoning process of a neural model.
  arXiv  Detail & Related papers  (2020-10-24T02:05:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.