Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learning

• URL: http://arxiv.org/abs/2204.07932v1
• Date: Sun, 17 Apr 2022 05:15:51 GMT
• Title: Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learning
• Authors: Jun Guo, Yonghong Chen, Yihang Hao, Zixin Yin, Yin Yu, Simin Li
• Abstract summary: It is crucial to test the robustness of c-MARL algorithm before it was deployed in reality. Existing adversarial attacks for MARL could be used for testing, but is limited to one robustness aspect. We propose MARLSafe, the first robustness testing framework for c-MARL algorithms.
• Score: 10.132303690998523
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While deep neural networks (DNNs) have strengthened the performance of cooperative multi-agent reinforcement learning (c-MARL), the agent policy can be easily perturbed by adversarial examples. Considering the safety critical applications of c-MARL, such as traffic management, power management and unmanned aerial vehicle control, it is crucial to test the robustness of c-MARL algorithm before it was deployed in reality. Existing adversarial attacks for MARL could be used for testing, but is limited to one robustness aspects (e.g., reward, state, action), while c-MARL model could be attacked from any aspect. To overcome the challenge, we propose MARLSafe, the first robustness testing framework for c-MARL algorithms. First, motivated by Markov Decision Process (MDP), MARLSafe consider the robustness of c-MARL algorithms comprehensively from three aspects, namely state robustness, action robustness and reward robustness. Any c-MARL algorithm must simultaneously satisfy these robustness aspects to be considered secure. Second, due to the scarceness of c-MARL attack, we propose c-MARL attacks as robustness testing algorithms from multiple aspects. Experiments on \textit{SMAC} environment reveals that many state-of-the-art c-MARL algorithms are of low robustness in all aspect, pointing out the urgent need to test and enhance robustness of c-MARL algorithms.

Related papers

• Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibrium [6.169364905804677]
  Multi-agent reinforcement learning (MARL) has achieved notable success in cooperative tasks. deploying MARL agents in real-world applications presents critical safety challenges. We propose a novel theoretical framework for safe MARL with $textitstate-wise$ constraints, where safety requirements are enforced at every state the agents visit. For practical deployment in complex high-dimensional systems, we propose $textitMulti-Agent Dual Actor-Critic$ (MADAC)
  arXiv  Detail & Related papers  (2024-11-22T16:08:42Z)
• Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Driving [3.5293763645151404]
  We propose a safe MARL method grounded in a Stackelberg model with bi-level optimization. We develop two practical algorithms, namely Constrained Stackelberg Q-learning (CSQ) and Constrained Stackelberg Multi-Agent Deep Deterministic Policy Gradient (CS-MADDPG) Our algorithms, CSQ and CS-MADDPG, outperform several strong MARL baselines, such as Bi-AC, MACPO, and MAPPO-L, regarding reward and safety performance.
  arXiv  Detail & Related papers  (2024-05-28T14:15:18Z)
• Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms [79.61176746380718]
  Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. MARL policies often lack robustness and are sensitive to small changes in their environment. We show that we can gain robustness by controlling a policy's Lipschitz constant. We propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies.
  arXiv  Detail & Related papers  (2023-10-16T20:14:06Z)
• Robust Multi-Agent Reinforcement Learning with State Uncertainty [17.916400875478377]
  We study the problem of MARL with state uncertainty in this work. We propose a robust multi-agent Q-learning algorithm to find such an equilibrium. Our experiments show that the proposed RMAQ algorithm converges to the optimal value function.
  arXiv  Detail & Related papers  (2023-07-30T12:31:42Z)
• Maximum Entropy Heterogeneous-Agent Reinforcement Learning [47.652866966384586]
  Multi-agent reinforcement learning (MARL) has been shown effective for cooperative games in recent years. We propose a unified framework for learning emphstochastic policies to resolve these issues. Based on the MaxEnt framework, we propose Heterogeneous-Agent Soft Actor-Critic (HASAC) algorithm.
  arXiv  Detail & Related papers  (2023-06-19T06:22:02Z)
• Provably Learning Nash Policies in Constrained Markov Potential Games [90.87573337770293]
  Multi-agent reinforcement learning (MARL) addresses sequential decision-making problems with multiple agents. Constrained Markov Games (CMGs) are a natural formalism for safe MARL problems, though generally intractable.
  arXiv  Detail & Related papers  (2023-06-13T13:08:31Z)
• Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning [17.957644784944755]
  We propose a novel certification method for c-MARLs to determine actions with guaranteed certified bounds. We empirically show that our certification bounds are much tighter than state-of-the-art RL certification solutions. Our method produces meaningful guaranteed robustness for all models and environments.
  arXiv  Detail & Related papers  (2022-12-22T14:36:27Z)
• Solving Continuous Control via Q-learning [54.05120662838286]
  We show that a simple modification of deep Q-learning largely alleviates issues with actor-critic methods. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods.
  arXiv  Detail & Related papers  (2022-10-22T22:55:50Z)
• MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library [82.77446613763809]
  We present MARLlib, a library designed to offer fast development for multi-agent tasks and algorithm combinations. MARLlib can effectively disentangle the intertwined nature of the multi-agent task and the learning process of the algorithm. The library's source code is publicly accessible on GitHub.
  arXiv  Detail & Related papers  (2022-10-11T03:11:12Z)
• Heterogeneous-Agent Mirror Learning: A Continuum of Solutions to Cooperative MARL [10.681450002239355]
  Heterogeneous-Agent Mirror Learning (HAML) provides a general template for MARL algorithmic designs. We prove that algorithms derived from the HAML template satisfy the desired properties of the monotonic improvement of the joint reward. We propose HAML extensions of two well-known RL algorithms, HAA2C (for A2C) and HADDPG (for DDPG)
  arXiv  Detail & Related papers  (2022-08-02T18:16:42Z)
• Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning [135.64775986546505]
  We exploit the symmetry of agents in multi-agent reinforcement learning (MARL) We propose MF-FQI algorithm that solves the mean-field MARL and establishes a non-asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.
  arXiv  Detail & Related papers  (2020-06-21T21:45:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.