Learning Causal Models of Autonomous Agents using Interventions
- URL: http://arxiv.org/abs/2108.09586v1
- Date: Sat, 21 Aug 2021 21:33:26 GMT
- Title: Learning Causal Models of Autonomous Agents using Interventions
- Authors: Pulkit Verma, Siddharth Srivastava
- Abstract summary: We extend the analysis of an agent assessment module that lets an AI system execute high-level instruction sequences in simulators.
We show that such a primitive query-response capability is sufficient to efficiently derive a user-interpretable causal model of the system.
- Score: 11.351235628684252
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: One of the several obstacles in the widespread use of AI systems is the lack
of requirements of interpretability that can enable a layperson to ensure the
safe and reliable behavior of such systems. We extend the analysis of an agent
assessment module that lets an AI system execute high-level instruction
sequences in simulators and answer the user queries about its execution of
sequences of actions. We show that such a primitive query-response capability
is sufficient to efficiently derive a user-interpretable causal model of the
system in stationary, fully observable, and deterministic settings. We also
introduce dynamic causal decision networks (DCDNs) that capture the causal
structure of STRIPS-like domains. A comparative analysis of different classes
of queries is also presented in terms of the computational requirements needed
to answer them and the efforts required to evaluate their responses to learn
the correct model.
Related papers
- Analyzing Adversarial Inputs in Deep Reinforcement Learning [53.3760591018817]
We present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification.
We introduce a novel metric, the Adversarial Rate, to classify models based on their susceptibility to such perturbations.
Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations.
arXiv Detail & Related papers (2024-02-07T21:58:40Z) - Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning [50.47568731994238]
Key method for creating Artificial Intelligence (AI) agents is Reinforcement Learning (RL)
This paper presents a general framework model for integrating and learning structured reasoning into AI agents' policies.
arXiv Detail & Related papers (2023-12-22T17:57:57Z) - Interactive Autonomous Navigation with Internal State Inference and
Interactivity Estimation [58.21683603243387]
We propose three auxiliary tasks with relational-temporal reasoning and integrate them into the standard Deep Learning framework.
These auxiliary tasks provide additional supervision signals to infer the behavior patterns other interactive agents.
Our approach achieves robust and state-of-the-art performance in terms of standard evaluation metrics.
arXiv Detail & Related papers (2023-11-27T18:57:42Z) - Towards Explainable AI for Channel Estimation in Wireless Communications [1.0874597293913013]
The aim of the proposed XAI-CHEST scheme is to identify the relevant model inputs by inducing high noise on the irrelevant ones.
As a result, the behavior of the studied DL-based channel estimators can be further analyzed and evaluated.
arXiv Detail & Related papers (2023-07-03T11:51:00Z) - Autonomous Capability Assessment of Sequential Decision-Making Systems
in Stochastic Settings (Extended Version) [27.825419721676766]
It is essential for users to understand what their AI systems can and can't do in order to use them safely.
This paper presents a new approach for modeling the capabilities of black-box AI systems that can plan and act.
arXiv Detail & Related papers (2023-06-07T22:05:48Z) - Causal Disentangled Variational Auto-Encoder for Preference
Understanding in Recommendation [50.93536377097659]
This paper introduces the Causal Disentangled Variational Auto-Encoder (CaD-VAE), a novel approach for learning causal disentangled representations from interaction data in recommender systems.
The approach utilizes structural causal models to generate causal representations that describe the causal relationship between latent factors.
arXiv Detail & Related papers (2023-04-17T00:10:56Z) - DeforestVis: Behavior Analysis of Machine Learning Models with Surrogate Decision Stumps [46.58231605323107]
We propose DeforestVis, a visual analytics tool that offers summarization of the behaviour of complex ML models.
DeforestVis helps users to explore the complexity versus fidelity trade-off by incrementally generating more stumps.
We show the applicability and usefulness of DeforestVis with two use cases and expert interviews with data analysts and model developers.
arXiv Detail & Related papers (2023-03-31T21:17:15Z) - Neural Causal Models for Counterfactual Identification and Estimation [62.30444687707919]
We study the evaluation of counterfactual statements through neural models.
First, we show that neural causal models (NCMs) are expressive enough.
Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions.
arXiv Detail & Related papers (2022-09-30T18:29:09Z) - Learning User-Interpretable Descriptions of Black-Box AI System
Capabilities [9.608555640607731]
This paper presents an approach for learning user-interpretable symbolic descriptions of the limits and capabilities of a black-box AI system.
It uses a hierarchical active querying paradigm to generate questions and to learn a user-interpretable model of the AI system based on its responses.
arXiv Detail & Related papers (2021-07-28T23:33:31Z) - i-Algebra: Towards Interactive Interpretability of Deep Neural Networks [41.13047686374529]
We present i-Algebra, a first-of-its-kind interactive framework for interpreting deep neural networks (DNNs)
At its core is a library of atomic, composable operators, which explain model behaviors at varying input granularity, during different inference stages, and from distinct interpretation perspectives.
We conduct user studies in a set of representative analysis tasks, including inspecting adversarial inputs, resolving model inconsistency, and cleansing contaminated data, all demonstrating its promising usability.
arXiv Detail & Related papers (2021-01-22T19:22:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.