Beware of "Explanations" of AI

• URL: http://arxiv.org/abs/2504.06791v1
• Date: Wed, 09 Apr 2025 11:31:08 GMT
• Title: Beware of "Explanations" of AI
• Authors: David Martens, Galit Shmueli, Theodoros Evgeniou, Kevin Bauer, Christian Janiesch, Stefan Feuerriegel, Sebastian Gabel, Sofie Goethals, Travis Greene, Nadja Klein, Mathias Kraus, Niklas Kühl, Claudia Perlich, Wouter Verbeke, Alona Zharova, Patrick Zschech, Foster Provost,
• Abstract summary: Understanding the decisions made and actions taken by increasingly complex AI system remains a key challenge.<n>This has led to an expanding field of research in explainable artificial intelligence (XAI)<n>The question of what constitutes a "good" explanation is dependent on the goals, stakeholders, and context.
• Score: 16.314859121110945
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the decisions made and actions taken by increasingly complex AI system remains a key challenge. This has led to an expanding field of research in explainable artificial intelligence (XAI), highlighting the potential of explanations to enhance trust, support adoption, and meet regulatory standards. However, the question of what constitutes a "good" explanation is dependent on the goals, stakeholders, and context. At a high level, psychological insights such as the concept of mental model alignment can offer guidance, but success in practice is challenging due to social and technical factors. As a result of this ill-defined nature of the problem, explanations can be of poor quality (e.g. unfaithful, irrelevant, or incoherent), potentially leading to substantial risks. Instead of fostering trust and safety, poorly designed explanations can actually cause harm, including wrong decisions, privacy violations, manipulation, and even reduced AI adoption. Therefore, we caution stakeholders to beware of explanations of AI: while they can be vital, they are not automatically a remedy for transparency or responsible AI adoption, and their misuse or limitations can exacerbate harm. Attention to these caveats can help guide future research to improve the quality and impact of AI explanations.

Related papers

• Imagining and building wise machines: The centrality of AI metacognition [78.76893632793497]
  We argue that shortcomings stem from one overarching failure: AI systems lack wisdom. While AI research has focused on task-level strategies, metacognition is underdeveloped in AI systems. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety.
  arXiv  Detail & Related papers  (2024-11-04T18:10:10Z)
• Combining AI Control Systems and Human Decision Support via Robustness and Criticality [53.10194953873209]
  We extend a methodology for adversarial explanations (AE) to state-of-the-art reinforcement learning frameworks. We show that the learned AI control system demonstrates robustness against adversarial tampering. In a training / learning framework, this technology can improve both the AI's decisions and explanations through human interaction.
  arXiv  Detail & Related papers  (2024-07-03T15:38:57Z)
• False Sense of Security in Explainable Artificial Intelligence (XAI) [3.298597939573779]
  We argue that AI regulations and current market conditions threaten effective AI governance and safety. Unless governments explicitly tackle the issue of explainability through clear legislative and policy statements, AI governance risks becoming a vacuous "box-ticking" exercise.
  arXiv  Detail & Related papers  (2024-05-06T20:02:07Z)
• VerifAI: Verified Generative AI [22.14231506649365]
  Generative AI has made significant strides, yet concerns about its accuracy and reliability continue to grow. We propose that verifying the outputs of generative AI from a data management perspective is an emerging issue for generative AI. Our vision is to promote the development of verifiable generative AI and contribute to a more trustworthy and responsible use of AI.
  arXiv  Detail & Related papers  (2023-07-06T06:11:51Z)
• It is not "accuracy vs. explainability" -- we need both for trustworthy AI systems [0.0]
  We are witnessing the emergence of an AI economy and society where AI technologies are increasingly impacting health care, business, transportation and many aspects of everyday life. However, AI systems may produce errors, can exhibit bias, may be sensitive to noise in the data, and often lack technical and judicial transparency resulting in reduction in trust and challenges in their adoption. These recent shortcomings and concerns have been documented in scientific but also in general press such as accidents with self driving cars, biases in healthcare, hiring and face recognition systems for people of color, seemingly correct medical decisions later found to be made due to wrong reasons etc.
  arXiv  Detail & Related papers  (2022-12-16T23:33:10Z)
• Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
  Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches. We explore this process with 43 AI practitioners and real end-users.
  arXiv  Detail & Related papers  (2022-11-12T21:54:05Z)
• Cybertrust: From Explainable to Actionable and Interpretable AI (AI2) [58.981120701284816]
  Actionable and Interpretable AI (AI2) will incorporate explicit quantifications and visualizations of user confidence in AI recommendations. It will allow examining and testing of AI system predictions to establish a basis for trust in the systems' decision making.
  arXiv  Detail & Related papers  (2022-01-26T18:53:09Z)
• Knowledge-intensive Language Understanding for Explainable AI [9.541228711585886]
  How AI-led decisions are made and what determining factors were included are crucial to understand. It is critical to have human-centered explanations that are directly related to decision-making. It is necessary to involve explicit domain knowledge that humans understand and use.
  arXiv  Detail & Related papers  (2021-08-02T21:12:30Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)
• Trustworthy AI: A Computational Perspective [54.80482955088197]
  We focus on six of the most crucial dimensions in achieving trustworthy AI: (i) Safety & Robustness, (ii) Non-discrimination & Fairness, (iii) Explainability, (iv) Privacy, (v) Accountability & Auditability, and (vi) Environmental Well-Being. For each dimension, we review the recent related technologies according to a taxonomy and summarize their applications in real-world systems.
  arXiv  Detail & Related papers  (2021-07-12T14:21:46Z)
• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies [1.2762298148425795]
  Lack of transparency is identified as one of the main barriers to implementation of AI systems in health care. We review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems. We conclude that explainable modelling can contribute to trustworthy AI, but the benefits of explainability still need to be proven in practice.
  arXiv  Detail & Related papers  (2020-07-31T09:08:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.