Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program

• URL: http://arxiv.org/abs/2106.05506v1
• Date: Thu, 10 Jun 2021 05:21:10 GMT
• Title: Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program
• Authors: Jeff Druce, James Niehaus, Vanessa Moody, David Jensen, Michael L. Littman
• Abstract summary: Deep neural network driven models have surpassed human level performance in benchmark autonomy tasks. The underlying policies for these agents, however, are not easily interpretable. This paper discusses the origins of these takeaways, provides amplifying information, and suggestions for future work.
• Score: 17.52385105997044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The advances in artificial intelligence enabled by deep learning architectures are undeniable. In several cases, deep neural network driven models have surpassed human level performance in benchmark autonomy tasks. The underlying policies for these agents, however, are not easily interpretable. In fact, given their underlying deep models, it is impossible to directly understand the mapping from observations to actions for any reasonably complex agent. Producing this supporting technology to "open the black box" of these AI systems, while not sacrificing performance, was the fundamental goal of the DARPA XAI program. In our journey through this program, we have several "big picture" takeaways: 1) Explanations need to be highly tailored to their scenario; 2) many seemingly high performing RL agents are extremely brittle and are not amendable to explanation; 3) causal models allow for rich explanations, but how to present them isn't always straightforward; and 4) human subjects conjure fantastically wrong mental models for AIs, and these models are often hard to break. This paper discusses the origins of these takeaways, provides amplifying information, and suggestions for future work.

Related papers

• Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning [53.45295657891099]
  This paper proposes Visual-O1, a multi-modal multi-turn chain-of-thought reasoning framework. It simulates human multi-modal multi-turn reasoning, providing instantial experience for highly intelligent models. Our work highlights the potential of artificial intelligence to work like humans in real-world scenarios with uncertainty and ambiguity.
  arXiv  Detail & Related papers  (2024-10-04T11:18:41Z)
• Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning [50.47568731994238]
  Key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL) This paper presents a general framework model for integrating and learning structured reasoning into AI agents' policies.
  arXiv  Detail & Related papers  (2023-12-22T17:57:57Z)
• Exploration with Principles for Diverse AI Supervision [88.61687950039662]
  Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. We propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data.
  arXiv  Detail & Related papers  (2023-10-13T07:03:39Z)
• Towards Reconciling Usability and Usefulness of Explainable AI Methodologies [2.715884199292287]
  Black-box AI systems can lead to liability and accountability issues when they produce an incorrect decision. Explainable AI (XAI) seeks to bridge the knowledge gap, between developers and end-users.
  arXiv  Detail & Related papers  (2023-01-13T01:08:49Z)
• Explainability Via Causal Self-Talk [9.149689942389923]
  Explaining the behavior of AI systems is an important problem that, in practice, is generally avoided. We describe an effective way to satisfy all the desiderata: train the AI system to build a causal model of itself. We implement this method in a simulated 3D environment, and show how it enables agents to generate faithful and semantically-meaningful explanations.
  arXiv  Detail & Related papers  (2022-11-17T23:17:01Z)
• Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
  Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches. We explore this process with 43 AI practitioners and real end-users.
  arXiv  Detail & Related papers  (2022-11-12T21:54:05Z)
• Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey [1.7205106391379026]
  Black-box nature of Artificial Intelligence (AI) models do not allow users to comprehend and sometimes trust the output created by such model. In AI applications, where not only the results but also the decision paths to the results are critical, such black-box AI models are not sufficient. Explainable Artificial Intelligence (XAI) addresses this problem and defines a set of AI models that are interpretable by the users.
  arXiv  Detail & Related papers  (2022-06-07T08:22:30Z)
• XAI for Cybersecurity: State of the Art, Challenges, Open Issues and Future Directions [16.633632244131775]
  AI models often appear as a blackbox wherein developers are unable to explain or trace back the reasoning behind a specific decision. Explainable AI (XAI) is a rapid growing field of research which helps to extract information and also visualize the results. The paper provides a brief overview on cybersecurity and the various forms of attack. Then the use of traditional AI techniques and its associated challenges are discussed which opens its doors towards use of XAI in various applications.
  arXiv  Detail & Related papers  (2022-06-03T02:15:30Z)
• Thinking Fast and Slow in AI: the Role of Metacognition [35.114607887343105]
  State-of-the-art AI still lacks many capabilities that would naturally be included in a notion of (human) intelligence. We argue that a better study of the mechanisms that allow humans to have these capabilities can help us understand how to imbue AI systems with these competencies.
  arXiv  Detail & Related papers  (2021-10-05T06:05:38Z)
• A User-Centred Framework for Explainable Artificial Intelligence in Human-Robot Interaction [70.11080854486953]
  We propose a user-centred framework for XAI that focuses on its social-interactive aspect. The framework aims to provide a structure for interactive XAI solutions thought for non-expert users.
  arXiv  Detail & Related papers  (2021-09-27T09:56:23Z)
• Explainability in Deep Reinforcement Learning [68.8204255655161]
  We review recent works in the direction to attain Explainable Reinforcement Learning (XRL) In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
  arXiv  Detail & Related papers  (2020-08-15T10:11:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.