Learning User-Interpretable Descriptions of Black-Box AI System Capabilities

• URL: http://arxiv.org/abs/2107.13668v1
• Date: Wed, 28 Jul 2021 23:33:31 GMT
• Title: Learning User-Interpretable Descriptions of Black-Box AI System Capabilities
• Authors: Pulkit Verma, Shashank Rao Marpally, Siddharth Srivastava
• Abstract summary: This paper presents an approach for learning user-interpretable symbolic descriptions of the limits and capabilities of a black-box AI system. It uses a hierarchical active querying paradigm to generate questions and to learn a user-interpretable model of the AI system based on its responses.
• Score: 9.608555640607731
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Several approaches have been developed to answer specific questions that a user may have about an AI system that can plan and act. However, the problems of identifying which questions to ask and that of computing a user-interpretable symbolic description of the overall capabilities of the system have remained largely unaddressed. This paper presents an approach for addressing these problems by learning user-interpretable symbolic descriptions of the limits and capabilities of a black-box AI system using low-level simulators. It uses a hierarchical active querying paradigm to generate questions and to learn a user-interpretable model of the AI system based on its responses. In contrast to prior work, we consider settings where imprecision of the user's conceptual vocabulary precludes a direct expression of the agent's capabilities. Furthermore, our approach does not require assumptions about the internal design of the target AI system or about the methods that it may use to compute or learn task solutions. Empirical evaluation on several game-based simulator domains shows that this approach can efficiently learn symbolic models of AI systems that use a deterministic black-box policy in fully observable scenarios.

Related papers

• Knowledge Tagging System on Math Questions via LLMs with Flexible Demonstration Retriever [48.5585921817745]
  Large Language Models (LLMs) are used to automate the knowledge tagging task. We show the strong performance of zero- and few-shot results over math questions knowledge tagging tasks. By proposing a reinforcement learning-based demonstration retriever, we successfully exploit the great potential of different-sized LLMs.
  arXiv  Detail & Related papers  (2024-06-19T23:30:01Z)
• Automated Process Planning Based on a Semantic Capability Model and SMT [50.76251195257306]
  In research of manufacturing systems and autonomous robots, the term capability is used for a machine-interpretable specification of a system function. We present an approach that combines these two topics: starting from a semantic capability model, an AI planning problem is automatically generated.
  arXiv  Detail & Related papers  (2023-12-14T10:37:34Z)
• Evaluating the Utility of Model Explanations for Model Development [54.23538543168767]
  We evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. To our surprise, we did not find evidence of significant improvement on tasks when users were provided with any of the saliency maps. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.
  arXiv  Detail & Related papers  (2023-12-10T23:13:23Z)
• Autonomous Capability Assessment of Sequential Decision-Making Systems in Stochastic Settings (Extended Version) [27.825419721676766]
  It is essential for users to understand what their AI systems can and can't do in order to use them safely. This paper presents a new approach for modeling the capabilities of black-box AI systems that can plan and act.
  arXiv  Detail & Related papers  (2023-06-07T22:05:48Z)
• Towards Human Cognition Level-based Experiment Design for Counterfactual Explanations (XAI) [68.8204255655161]
  The emphasis of XAI research appears to have turned to a more pragmatic explanation approach for better understanding. An extensive area where cognitive science research may substantially influence XAI advancements is evaluating user knowledge and feedback. We propose a framework to experiment with generating and evaluating the explanations on the grounds of different cognitive levels of understanding.
  arXiv  Detail & Related papers  (2022-10-31T19:20:22Z)
• Learning Causal Models of Autonomous Agents using Interventions [11.351235628684252]
  We extend the analysis of an agent assessment module that lets an AI system execute high-level instruction sequences in simulators. We show that such a primitive query-response capability is sufficient to efficiently derive a user-interpretable causal model of the system.
  arXiv  Detail & Related papers  (2021-08-21T21:33:26Z)
• Explaining Black-Box Algorithms Using Probabilistic Contrastive Counterfactuals [7.727206277914709]
  We propose a principled causality-based approach for explaining black-box decision-making systems. We show how such counterfactuals can quantify the direct and indirect influences of a variable on decisions made by an algorithm. We show how such counterfactuals can provide actionable recourse for individuals negatively affected by the algorithm's decision.
  arXiv  Detail & Related papers  (2021-03-22T16:20:21Z)
• Explanation Ontology: A Model of Explanations for User-Centered AI [3.1783442097247345]
  Explanations have often added to an AI system in a non-principled, post-hoc manner. With greater adoption of these systems and emphasis on user-centric explainability, there is a need for a structured representation that treats explainability as a primary consideration. We design an explanation ontology to model both the role of explanations, accounting for the system and user attributes in the process, and the range of different literature-derived explanation types.
  arXiv  Detail & Related papers  (2020-10-04T03:53:35Z)
• Towards an Interface Description Template for AI-enabled Systems [77.34726150561087]
  Reuse is a common system architecture approach that seeks to instantiate a system architecture with existing components. There is currently no framework that guides the selection of necessary information to assess their portability to operate in a system different than the one for which the component was originally purposed. We present ongoing work on establishing an interface description template that captures the main information of an AI-enabled component.
  arXiv  Detail & Related papers  (2020-07-13T20:30:26Z)
• A general framework for scientifically inspired explanations in AI [76.48625630211943]
  We instantiate the concept of structure of scientific explanation as the theoretical underpinning for a general framework in which explanations for AI systems can be implemented. This framework aims to provide the tools to build a "mental-model" of any AI system so that the interaction with the user can provide information on demand and be closer to the nature of human-made explanations.
  arXiv  Detail & Related papers  (2020-03-02T10:32:21Z)
• Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations [44.016120003139264]
  This paper introduces methods for providing contrastive explanations in terms of user-specified concepts for sequential decision-making settings. We do this by building partial symbolic models of a local approximation of the task that can be leveraged to answer the user queries.
  arXiv  Detail & Related papers  (2020-02-04T01:37:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.