Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2302.02180v2
• Date: Wed, 17 May 2023 03:19:08 GMT
• Title: Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learning
• Authors: Zhiwei Xu, Bin Zhang, Dapeng Li, Guangchong Zhou, Zeren Zhang, Guoliang Fan
• Abstract summary: We propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.
• Score: 12.74348597962689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Value decomposition methods have gained popularity in the field of cooperative multi-agent reinforcement learning. However, almost all existing methods follow the principle of Individual Global Max (IGM) or its variants, which limits their problem-solving capabilities. To address this, we propose a dual self-awareness value decomposition framework, inspired by the notion of dual self-awareness in psychology, that entirely rejects the IGM premise. Each agent consists of an ego policy for action selection and an alter ego value function to solve the credit assignment problem. The value function factorization can ignore the IGM assumption by utilizing an explicit search procedure. On the basis of the above, we also suggest a novel anti-ego exploration mechanism to avoid the algorithm becoming stuck in a local optimum. As the first fully IGM-free value decomposition method, our proposed framework achieves desirable performance in various cooperative tasks.

Related papers

• ME-IGM: Individual-Global-Max in Maximum Entropy Multi-Agent Reinforcement Learning [9.55036711735232]
  We introduce ME-IGM, a novel maximum entropy MARL algorithm compatible with any credit assignment mechanism.<n>We demonstrate its state-of-the-art performance across 17 scenarios in SMAC-v2 and Overcooked.
  arXiv  Detail & Related papers  (2024-06-20T01:55:08Z)
• Reaching Consensus in Cooperative Multi-Agent Reinforcement Learning with Goal Imagination [16.74629849552254]
  We propose a model-based consensus mechanism to explicitly coordinate multiple agents. The proposed Multi-agent Goal Imagination (MAGI) framework guides agents to reach consensus with an Imagined common goal. We show that such efficient consensus mechanism can guide all agents cooperatively reaching valuable future states.
  arXiv  Detail & Related papers  (2024-03-05T18:07:34Z)
• Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applications [73.58451824894568]
  The widely adopted CMDP model constrains the risks in expectation, which makes room for dangerous behaviors in long-tail states. In safety-critical domains, such behaviors could lead to disastrous outcomes. We propose Objective Suppression, a novel method that adaptively suppresses the task reward maximizing objectives according to a safety critic.
  arXiv  Detail & Related papers  (2024-02-23T23:22:06Z)
• Robust and Performance Incentivizing Algorithms for Multi-Armed Bandits with Strategic Agents [52.75161794035767]
  We introduce a class of bandit algorithms that meet the two objectives of performance incentivization and robustness simultaneously. We show that settings where the principal has no information about the arms' performance characteristics can be handled by combining ideas from second price auctions with our algorithms.
  arXiv  Detail & Related papers  (2023-12-13T06:54:49Z)
• QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning [2.287186762346021]
  We propose QFree, a universal value function factorization method for multi-agent reinforcement learning. We show that QFree achieves the state-of-the-art performance in a general-purpose complex MARL benchmark environment.
  arXiv  Detail & Related papers  (2023-11-01T08:07:16Z)
• Achieving Fairness in Multi-Agent Markov Decision Processes Using Reinforcement Learning [30.605881670761853]
  We propose a Reinforcement Learning approach to achieve fairness in finite-horizon episodic MDPs. We show that such an approach achieves sub-linear regret in terms of the number of episodes.
  arXiv  Detail & Related papers  (2023-06-01T03:43:53Z)
• Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning [48.41925886860991]
  Real-world cooperation often requires intensive coordination among agents simultaneously. Traditional methods that learn the value function as a monotonic mixing of per-agent utilities cannot solve the tasks with non-monotonic returns. We propose a novel explicit credit assignment method to address the non-monotonic problem.
  arXiv  Detail & Related papers  (2023-02-14T07:23:59Z)
• PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning [43.862956745961654]
  Multi-agent reinforcement learning (MARL) has witnessed significant progress with the development of value function factorization methods. In this paper, we show that in partially observable MARL problems, an agent's ordering over its own actions could impose concurrent constraints. We propose PAC, a new framework leveraging information generated from Counterfactual Predictions of optimal joint action selection.
  arXiv  Detail & Related papers  (2022-06-22T23:34:30Z)
• Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation [55.96893934962757]
  In multi-agent system, polices of different agents need to be evaluated jointly. In current methods, value functions or advantage functions use counter-factual joint actions which are evaluated asynchronously. In this work, we propose the approximatively synchronous advantage estimation.
  arXiv  Detail & Related papers  (2020-12-07T07:29:19Z)
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning [53.73686229912562]
  We propose a novel MARL approach called Universal Value Exploration (UneVEn) UneVEn learns a set of related tasks simultaneously with a linear decomposition of universal successor features. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail.
  arXiv  Detail & Related papers  (2020-10-06T19:08:47Z)
• VCG Mechanism Design with Unknown Agent Values under Stochastic Bandit Feedback [104.06766271716774]
  We study a multi-round welfare-maximising mechanism design problem in instances where agents do not know their values. We first define three notions of regret for the welfare, the individual utilities of each agent and that of the mechanism. Our framework also provides flexibility to control the pricing scheme so as to trade-off between the agent and seller regrets.
  arXiv  Detail & Related papers  (2020-04-19T18:00:58Z)
• FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC) It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2020-03-14T21:29:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.