Fundamental Limitation in Explaining AI

• URL: http://arxiv.org/abs/2605.24727v2
• Date: Sun, 31 May 2026 11:58:29 GMT
• Title: Fundamental Limitation in Explaining AI
• Abstract summary: We prove a quadrilemma in explaining AI, stating that AI and its explanation cannot satisfy the following four conditions simultaneously.<n>The quadrilemma implies that AI governance should be designed on the premise that the faithfulness of AI explanations is always incomplete.
• Score: 4.405551654403497
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While large-scale models such as LLMs and diffusion models have achieved practical success, public institutions have emphasized the importance of explainability in AI. Existing methods for explaining AI, however, are not designed to provide completely faithful explanations of the behavior of large-scale AI systems. Although a completely faithful and interpretable explanation of the behavior of an AI system might be useful for AI governance, it has not been known whether providing such an explanation is theoretically possible. In this paper, we mathematically prove a fundamental quadrilemma in explaining AI, stating that AI and its explanation cannot satisfy the following four conditions simultaneously: 1) the complexity of the operation environment, 2) the goodness of the AI's performance, 3) the interpretability of the AI's explanation, and 4) the complete faithfulness of the AI's explanation. This quadrilemma suggests that, in most applications where we cannot change the environment or sacrifice good AI performance and an interpretable explanation, we should give up complete faithfulness of explanations and should instead aim to explain only the parts that are important for applications. As a consequence, the quadrilemma implies that AI governance should be designed on the premise that the faithfulness of AI explanations is always incomplete.

Related papers

• Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor [83.99510317617694]
  We argue that a broader conception of what rigorous AI research and practice should entail is needed.<n>We aim to provide useful language and a framework for much-needed dialogue about the AI community's work.
  arXiv  Detail & Related papers  (2025-06-17T15:44:41Z)
• Explainable AI: Definition and attributes of a good explanation for health AI [0.18846515534317265]
  understanding how and why an AI system makes a recommendation may require complex explanations of its inner workings and reasoning processes. To fully realize the potential of AI, it is critical to address two fundamental questions about explanations for safety-critical AI applications. The research outputs include (1) a definition of what constitutes an explanation in health-AI and (2) a comprehensive list of attributes that characterize a good explanation in health-AI.
  arXiv  Detail & Related papers  (2024-09-09T16:56:31Z)
• On the consistent reasoning paradox of intelligence and optimal trust in AI: The power of 'I don't know' [79.69412622010249]
  Consistent reasoning, which lies at the core of human intelligence, is the ability to handle tasks that are equivalent. CRP asserts that consistent reasoning implies fallibility -- in particular, human-like intelligence in AI necessarily comes with human-like fallibility.
  arXiv  Detail & Related papers  (2024-08-05T10:06:53Z)
• In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making [25.18203172421461]
  We argue explanations are only useful to the extent that they allow a human decision maker to verify the correctness of an AI's prediction. We also compare the objective of complementary performance with that of appropriate reliance, decomposing the latter into the notions of outcome-graded and strategy-graded reliance.
  arXiv  Detail & Related papers  (2023-05-12T18:28:04Z)
• Alterfactual Explanations -- The Relevance of Irrelevance for Explaining AI Systems [0.9542023122304099]
  We argue that in order to fully understand a decision, not only knowledge about relevant features is needed, but that the awareness of irrelevant information also highly contributes to the creation of a user's mental model of an AI system. Our approach, which we call Alterfactual Explanations, is based on showing an alternative reality where irrelevant features of an AI's input are altered. We show that alterfactual explanations are suited to convey an understanding of different aspects of the AI's reasoning than established counterfactual explanation methods.
  arXiv  Detail & Related papers  (2022-07-19T16:20:37Z)
• Cybertrust: From Explainable to Actionable and Interpretable AI (AI2) [58.981120701284816]
  Actionable and Interpretable AI (AI2) will incorporate explicit quantifications and visualizations of user confidence in AI recommendations. It will allow examining and testing of AI system predictions to establish a basis for trust in the systems' decision making.
  arXiv  Detail & Related papers  (2022-01-26T18:53:09Z)
• Scope and Sense of Explainability for AI-Systems [0.0]
  Emphasis will be given to difficulties related to the explainability of highly complex and efficient AI systems. It will be elaborated on arguments supporting the notion that if AI-solutions were to be discarded in advance because of their not being thoroughly comprehensible, a great deal of the potentiality of intelligent systems would be wasted.
  arXiv  Detail & Related papers  (2021-12-20T14:25:05Z)
• Some Critical and Ethical Perspectives on the Empirical Turn of AI Interpretability [0.0]
  We consider two issues currently faced by Artificial Intelligence development: the lack of ethics and interpretability of AI decisions. We experimentally show that the empirical and liberal turn of the production of explanations tends to select AI explanations with a low denunciatory power. We propose two scenarios for the future development of ethical AI: more external regulation or more liberalization of AI explanations.
  arXiv  Detail & Related papers  (2021-09-20T14:41:50Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)
• A general framework for scientifically inspired explanations in AI [76.48625630211943]
  We instantiate the concept of structure of scientific explanation as the theoretical underpinning for a general framework in which explanations for AI systems can be implemented. This framework aims to provide the tools to build a "mental-model" of any AI system so that the interaction with the user can provide information on demand and be closer to the nature of human-made explanations.
  arXiv  Detail & Related papers  (2020-03-02T10:32:21Z)
• Learning from Learning Machines: Optimisation, Rules, and Social Norms [91.3755431537592]
  It appears that the area of AI that is most analogous to the behaviour of economic entities is that of morally good decision-making. Recent successes of deep learning for AI suggest that more implicit specifications work better than explicit ones for solving such problems.
  arXiv  Detail & Related papers  (2019-12-29T17:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.