Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values

• URL: http://arxiv.org/abs/2110.01307v1
• Date: Mon, 4 Oct 2021 10:28:57 GMT
• Title: Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values
• Authors: Alexandre Heuillet, Fabien Couthouis and Natalia D\'iaz-Rodr\'iguez
• Abstract summary: This study proposes a novel approach to explain cooperative strategies in multiagent RL using Shapley values. Results could have implications for non-discriminatory decision making, ethical and responsible AI-derived decisions or policy making under fairness constraints.
• Score: 68.8204255655161
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Explainable Artificial Intelligence (XAI) is increasingly expanding more areas of application, little has been applied to make deep Reinforcement Learning (RL) more comprehensible. As RL becomes ubiquitous and used in critical and general public applications, it is essential to develop methods that make it better understood and more interpretable. This study proposes a novel approach to explain cooperative strategies in multiagent RL using Shapley values, a game theory concept used in XAI that successfully explains the rationale behind decisions taken by Machine Learning algorithms. Through testing common assumptions of this technique in two cooperation-centered socially challenging multi-agent environments environments, this article argues that Shapley values are a pertinent way to evaluate the contribution of players in a cooperative multi-agent RL context. To palliate the high overhead of this method, Shapley values are approximated using Monte Carlo sampling. Experimental results on Multiagent Particle and Sequential Social Dilemmas show that Shapley values succeed at estimating the contribution of each agent. These results could have implications that go beyond games in economics, (e.g., for non-discriminatory decision making, ethical and responsible AI-derived decisions or policy making under fairness constraints). They also expose how Shapley values only give general explanations about a model and cannot explain a single run, episode nor justify precise actions taken by agents. Future work should focus on addressing these critical aspects.

Related papers

• Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks [94.07688076435818]
  We study reinforcement learning for learning a Quantal Stackelberg Equilibrium (QSE) in an episodic Markov game with a leader-follower structure. Our algorithms are based on (i) learning the quantal response model via maximum likelihood estimation and (ii) model-free or model-based RL for solving the leader's decision making problem.
  arXiv  Detail & Related papers  (2023-07-26T10:24:17Z)
• Multi-agent Policy Reciprocity with Theoretical Guarantee [24.65151626601257]
  We propose a novel multi-agent policy reciprocity (PR) framework, where each agent can fully exploit cross-agent policies even in mismatched states. Experimental results on discrete and continuous environments demonstrate that PR outperforms various existing RL and transfer RL methods.
  arXiv  Detail & Related papers  (2023-04-12T06:27:10Z)
• MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning [62.065503126104126]
  We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen learning agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. We introduce MERMAIDE, a model-based meta-learning framework to train a principal that can quickly adapt to out-of-distribution agents.
  arXiv  Detail & Related papers  (2023-04-10T15:44:50Z)
• Towards a more efficient computation of individual attribute and policy contribution for post-hoc explanation of cooperative multi-agent systems using Myerson values [0.0]
  A quantitative assessment of the global importance of an agent in a team is as valuable as gold for strategists, decision-makers, and sports coaches. We propose a method to determine a Hierarchical Knowledge Graph of agents' policies and features in a Multi-Agent System. We test the proposed approach in a proof-of-case environment deploying both hardcoded policies and policies obtained via Deep Reinforcement Learning.
  arXiv  Detail & Related papers  (2022-12-06T15:15:00Z)
• Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning [85.86440477005523]
  We study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RIRL models the cost of cognitive information processing using mutual information. We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions.
  arXiv  Detail & Related papers  (2022-01-18T20:54:00Z)
• Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey [0.7366405857677226]
  Reinforcement Learning (RL) methods provide a potential backbone for the cognitive model required for the development of Broad-XAI. RL represents a suite of approaches that have had increasing success in solving a range of sequential decision-making problems. This paper aims to introduce a conceptual framework, called the Causal XRL Framework (CXF), that unifies the current XRL research and uses RL as a backbone to the development of Broad-XAI.
  arXiv  Detail & Related papers  (2021-08-20T05:18:50Z)
• Rational Shapley Values [0.0]
  Most popular tools for post-hoc explainable artificial intelligence (XAI) are either insensitive to context or difficult to summarize. I introduce emphrational Shapley values, a novel XAI method that synthesizes and extends these seemingly incompatible approaches. I leverage tools from decision theory and causal modeling to formalize and implement a pragmatic approach that resolves a number of known challenges in XAI.
  arXiv  Detail & Related papers  (2021-06-18T15:45:21Z)
• Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning [92.05556163518999]
  MARL exacerbates matters by imposing various constraints on communication and observability. For value-based methods, it poses challenges in accurately representing the optimal value function. For policy gradient methods, it makes training the critic difficult and exacerbates the problem of the lagging critic. We show that from a learning theory perspective, both problems can be addressed by accurately representing the associated action-value function.
  arXiv  Detail & Related papers  (2021-05-31T23:08:05Z)
• Explainability in Deep Reinforcement Learning [68.8204255655161]
  We review recent works in the direction to attain Explainable Reinforcement Learning (XRL) In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
  arXiv  Detail & Related papers  (2020-08-15T10:11:42Z)
• Explainable Reinforcement Learning: A Survey [0.0]
  Explainable Artificial Intelligence (XAI) has gained increased traction over the last few years. XAI models exhibit one detrimential characteristic: a performance-transparency trade-off. This survey attempts to address this gap by offering an overview of Explainable Reinforcement Learning (XRL) methods.
  arXiv  Detail & Related papers  (2020-05-13T10:52:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.