On the Robustness of Cooperative Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2003.03722v1
• Date: Sun, 8 Mar 2020 05:12:13 GMT
• Title: On the Robustness of Cooperative Multi-Agent Reinforcement Learning
• Authors: Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, Nicolas Papernot
• Abstract summary: In cooperative multi-agent reinforcement learning (c-MARL), agents learn to cooperatively take actions as a team to maximize a total team reward. We analyze the robustness of c-MARL to adversaries capable of attacking one of the agents on a team. By attacking a single agent, our attack method has highly negative impact on the overall team reward, reducing it from 20 to 9.4.
• Score: 32.92198917228515
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In cooperative multi-agent reinforcement learning (c-MARL), agents learn to cooperatively take actions as a team to maximize a total team reward. We analyze the robustness of c-MARL to adversaries capable of attacking one of the agents on a team. Through the ability to manipulate this agent's observations, the adversary seeks to decrease the total team reward. Attacking c-MARL is challenging for three reasons: first, it is difficult to estimate team rewards or how they are impacted by an agent mispredicting; second, models are non-differentiable; and third, the feature space is low-dimensional. Thus, we introduce a novel attack. The attacker first trains a policy network with reinforcement learning to find a wrong action it should encourage the victim agent to take. Then, the adversary uses targeted adversarial examples to force the victim to take this action. Our results on the StartCraft II multi-agent benchmark demonstrate that c-MARL teams are highly vulnerable to perturbations applied to one of their agent's observations. By attacking a single agent, our attack method has highly negative impact on the overall team reward, reducing it from 20 to 9.4. This results in the team's winning rate to go down from 98.9% to 0%.

Related papers

• Adversarial Inception for Bounded Backdoor Poisoning in Deep Reinforcement Learning [16.350898218047405]
  We propose a new class of backdoor attacks against Deep Reinforcement Learning (DRL) algorithms. These attacks achieve state of the art performance while minimally altering the agent's rewards. We then devise an online attack which significantly out-performs prior attacks under bounded reward constraints.
  arXiv  Detail & Related papers  (2024-10-17T19:50:28Z)
• CuDA2: An approach for Incorporating Traitor Agents into Cooperative Multi-Agent Systems [13.776447110639193]
  We introduce a novel method that involves injecting traitor agents into the CMARL system. In TMDP, traitors are trained using the same MARL algorithm as the victim agents, with their reward function set as the negative of the victim agents' reward. CuDA2 enhances the efficiency and aggressiveness of attacks on the specified victim agents' policies.
  arXiv  Detail & Related papers  (2024-06-25T09:59:31Z)
• Toward Optimal LLM Alignments Using Two-Player Games [86.39338084862324]
  In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.
  arXiv  Detail & Related papers  (2024-06-16T15:24:50Z)
• Malicious Agent Detection for Robust Multi-Agent Collaborative Perception [52.261231738242266]
  Multi-agent collaborative (MAC) perception is more vulnerable to adversarial attacks than single-agent perception. We propose Malicious Agent Detection (MADE), a reactive defense specific to MAC perception. We conduct comprehensive evaluations on a benchmark 3D dataset V2X-sim and a real-road dataset DAIR-V2X.
  arXiv  Detail & Related papers  (2023-10-18T11:36:42Z)
• Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning [45.408568528354216]
  We investigate the impact of adversarial attacks on multi-agent reinforcement learning (MARL) In the considered setup, there is an attacker who is able to modify the rewards before the agents receive them or manipulate the actions before the environment receives them. We show that the mixed attack strategy can efficiently attack MARL agents even if the attacker has no prior information about the underlying environment and the agents' algorithms.
  arXiv  Detail & Related papers  (2023-07-15T00:38:55Z)
• Attacking Cooperative Multi-Agent Reinforcement Learning by Adversarial Minority Influence [41.14664289570607]
  Adrial Minority Influence (AMI) is a practical black-box attack and can be launched without knowing victim parameters. AMI is also strong by considering the complex multi-agent interaction and the cooperative goal of agents. We achieve the first successful attack against real-world robot swarms and effectively fool agents in simulated environments into collectively worst-case scenarios.
  arXiv  Detail & Related papers  (2023-02-07T08:54:37Z)
• Widen The Backdoor To Let More Attackers In [24.540853975732922]
  We investigate the scenario of a multi-agent backdoor attack, where multiple non-colluding attackers craft and insert triggered samples in a shared dataset. We discover a clear backfiring phenomenon: increasing the number of attackers shrinks each attacker's attack success rate. We then exploit this phenomenon to minimize the collective ASR of attackers and maximize defender's robustness accuracy.
  arXiv  Detail & Related papers  (2021-10-09T13:53:57Z)
• BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning [80.99426477001619]
  We migrate backdoor attacks to more complex RL systems involving multiple agents. As a proof of concept, we demonstrate that an adversary agent can trigger the backdoor of the victim agent with its own action. The results show that when the backdoor is activated, the winning rate of the victim drops by 17% to 37% compared to when not activated.
  arXiv  Detail & Related papers  (2021-05-02T23:47:55Z)
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary [86.0846119254031]
  We study the robustness of reinforcement learning with adversarially perturbed state observations. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
  arXiv  Detail & Related papers  (2021-01-21T05:38:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.