Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning

• URL: http://arxiv.org/abs/2208.13663v1
• Date: Mon, 29 Aug 2022 15:10:14 GMT
• Title: Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning
• Authors: Anshuka Rangi, Haifeng Xu, Long Tran-Thanh, Massimo Franceschetti
• Abstract summary: This paper studies poisoning attacks to manipulate emphany order-optimal learning algorithm towards a targeted policy in episodic RL. We find that the effect of attacks crucially depend on whether the rewards are bounded or unbounded.
• Score: 36.30086280732181
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To understand the security threats to reinforcement learning (RL) algorithms, this paper studies poisoning attacks to manipulate \emph{any} order-optimal learning algorithm towards a targeted policy in episodic RL and examines the potential damage of two natural types of poisoning attacks, i.e., the manipulation of \emph{reward} and \emph{action}. We discover that the effect of attacks crucially depend on whether the rewards are bounded or unbounded. In bounded reward settings, we show that only reward manipulation or only action manipulation cannot guarantee a successful attack. However, by combining reward and action manipulation, the adversary can manipulate any order-optimal learning algorithm to follow any targeted policy with $\tilde{\Theta}(\sqrt{T})$ total attack cost, which is order-optimal, without any knowledge of the underlying MDP. In contrast, in unbounded reward settings, we show that reward manipulation attacks are sufficient for an adversary to successfully manipulate any order-optimal learning algorithm to follow any targeted policy using $\tilde{O}(\sqrt{T})$ amount of contamination. Our results reveal useful insights about what can or cannot be achieved by poisoning attacks, and are set to spur more works on the design of robust RL algorithms.

Related papers

• Optimal Attack and Defense for Reinforcement Learning [11.36770403327493]
  In adversarial RL, an external attacker has the power to manipulate the victim agent's interaction with the environment. We show the attacker's problem of designing a stealthy attack that maximizes its own expected reward. We argue that the optimal defense policy for the victim can be computed as the solution to a Stackelberg game.
  arXiv  Detail & Related papers  (2023-11-30T21:21:47Z)
• Adversarial Attacks on Adversarial Bandits [10.891819703383408]
  We show that the attacker is able to mislead any no-regret adversarial bandit algorithm into selecting a suboptimal target arm. This result implies critical security concern in real-world bandit-based systems.
  arXiv  Detail & Related papers  (2023-01-30T00:51:39Z)
• Projective Ranking-based GNN Evasion Attacks [52.85890533994233]
  Graph neural networks (GNNs) offer promising learning methods for graph-related tasks. GNNs are at risk of adversarial attacks.
  arXiv  Detail & Related papers  (2022-02-25T21:52:09Z)
• Adversarial Attacks on Gaussian Process Bandits [47.84198626686564]
  We propose various adversarial attack methods with differing assumptions on the attacker's strength and prior information. Our goal is to understand adversarial attacks on GP bandits from both a theoretical and practical perspective. We demonstrate that adversarial attacks on GP bandits can succeed in forcing the algorithm towards $mathcalR_rm target$ even with a low attack budget.
  arXiv  Detail & Related papers  (2021-10-16T02:39:10Z)
• Disturbing Reinforcement Learning Agents with Corrupted Rewards [62.997667081978825]
  We analyze the effects of different attack strategies based on reward perturbations on reinforcement learning algorithms. We show that smoothly crafting adversarial rewards are able to mislead the learner, and that using low exploration probability values, the policy learned is more robust to corrupt rewards.
  arXiv  Detail & Related papers  (2021-02-12T15:53:48Z)
• Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
  Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs. We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2020-08-05T07:49:42Z)
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning [43.07944714475278]
  In reward-poisoning attacks against reinforcement learning, an attacker can perturb the environment reward $r_t$ into $r_t+delta_t$ at each step. We show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps in state-space size $|S|$. We also show that an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.
  arXiv  Detail & Related papers  (2020-03-27T19:46:23Z)
• Action-Manipulation Attacks Against Stochastic Bandits: Attacks and Defense [45.408568528354216]
  We introduce a new class of attack named action-manipulation attack. In this attack, an adversary can change the action signal selected by the user. To defend against this class of attacks, we introduce a novel algorithm that is robust to action-manipulation attacks.
  arXiv  Detail & Related papers  (2020-02-19T04:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.