Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values

• URL: http://arxiv.org/abs/2110.01307v1
• Date: Mon, 4 Oct 2021 10:28:57 GMT
• Title: Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values
• Authors: Alexandre Heuillet, Fabien Couthouis and Natalia D\'iaz-Rodr\'iguez
• Abstract summary: This study proposes a novel approach to explain cooperative strategies in multiagent RL using Shapley values. Results could have implications for non-discriminatory decision making, ethical and responsible AI-derived decisions or policy making under fairness constraints.
• Score: 68.8204255655161
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Explainable Artificial Intelligence (XAI) is increasingly expanding more areas of application, little has been applied to make deep Reinforcement Learning (RL) more comprehensible. As RL becomes ubiquitous and used in critical and general public applications, it is essential to develop methods that make it better understood and more interpretable. This study proposes a novel approach to explain cooperative strategies in multiagent RL using Shapley values, a game theory concept used in XAI that successfully explains the rationale behind decisions taken by Machine Learning algorithms. Through testing common assumptions of this technique in two cooperation-centered socially challenging multi-agent environments environments, this article argues that Shapley values are a pertinent way to evaluate the contribution of players in a cooperative multi-agent RL context. To palliate the high overhead of this method, Shapley values are approximated using Monte Carlo sampling. Experimental results on Multiagent Particle and Sequential Social Dilemmas show that Shapley values succeed at estimating the contribution of each agent. These results could have implications that go beyond games in economics, (e.g., for non-discriminatory decision making, ethical and responsible AI-derived decisions or policy making under fairness constraints). They also expose how Shapley values only give general explanations about a model and cannot explain a single run, episode nor justify precise actions taken by agents. Future work should focus on addressing these critical aspects.

Related papers

• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Semifactual Explanations for Reinforcement Learning [1.5320737596132754]
  Reinforcement Learning (RL) is a learning paradigm in which the agent learns from its environment through trial and error. Deep reinforcement learning (DRL) algorithms represent the agent's policies using neural networks, making their decisions difficult to interpret. Explaining the behaviour of DRL agents is necessary to advance user trust, increase engagement, and facilitate integration with real-life tasks.
  arXiv  Detail & Related papers  (2024-09-09T08:37:47Z)
• Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contracts [20.8288955218712]
  We propose a framework where a principal guides an agent in a Markov Decision Process (MDP) using a series of contracts. We present and analyze a meta-algorithm that iteratively optimize the policies of the principal and agent. We then scale our algorithm with deep Q-learning and analyze its convergence in the presence of approximation error.
  arXiv  Detail & Related papers  (2024-07-25T14:28:58Z)
• Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks [94.07688076435818]
  We study reinforcement learning for learning a Quantal Stackelberg Equilibrium (QSE) in an episodic Markov game with a leader-follower structure. Our algorithms are based on (i) learning the quantal response model via maximum likelihood estimation and (ii) model-free or model-based RL for solving the leader's decision making problem.
  arXiv  Detail & Related papers  (2023-07-26T10:24:17Z)
• MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning [62.065503126104126]
  We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen learning agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. We introduce MERMAIDE, a model-based meta-learning framework to train a principal that can quickly adapt to out-of-distribution agents.
  arXiv  Detail & Related papers  (2023-04-10T15:44:50Z)
• Towards a more efficient computation of individual attribute and policy contribution for post-hoc explanation of cooperative multi-agent systems using Myerson values [0.0]
  A quantitative assessment of the global importance of an agent in a team is as valuable as gold for strategists, decision-makers, and sports coaches. We propose a method to determine a Hierarchical Knowledge Graph of agents' policies and features in a Multi-Agent System. We test the proposed approach in a proof-of-case environment deploying both hardcoded policies and policies obtained via Deep Reinforcement Learning.
  arXiv  Detail & Related papers  (2022-12-06T15:15:00Z)
• Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning [85.86440477005523]
  We study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RIRL models the cost of cognitive information processing using mutual information. We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions.
  arXiv  Detail & Related papers  (2022-01-18T20:54:00Z)
• Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey [0.7366405857677226]
  Reinforcement Learning (RL) methods provide a potential backbone for the cognitive model required for the development of Broad-XAI. RL represents a suite of approaches that have had increasing success in solving a range of sequential decision-making problems. This paper aims to introduce a conceptual framework, called the Causal XRL Framework (CXF), that unifies the current XRL research and uses RL as a backbone to the development of Broad-XAI.
  arXiv  Detail & Related papers  (2021-08-20T05:18:50Z)
• Rational Shapley Values [0.0]
  Most popular tools for post-hoc explainable artificial intelligence (XAI) are either insensitive to context or difficult to summarize. I introduce emphrational Shapley values, a novel XAI method that synthesizes and extends these seemingly incompatible approaches. I leverage tools from decision theory and causal modeling to formalize and implement a pragmatic approach that resolves a number of known challenges in XAI.
  arXiv  Detail & Related papers  (2021-06-18T15:45:21Z)
• Explainability in Deep Reinforcement Learning [68.8204255655161]
  We review recent works in the direction to attain Explainable Reinforcement Learning (XRL) In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
  arXiv  Detail & Related papers  (2020-08-15T10:11:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.