Manipulating Reinforcement Learning: Poisoning Attacks on Cost Signals

• URL: http://arxiv.org/abs/2002.03827v2
• Date: Mon, 20 Jul 2020 22:55:26 GMT
• Title: Manipulating Reinforcement Learning: Poisoning Attacks on Cost Signals
• Authors: Yunhan Huang and Quanyan Zhu
• Abstract summary: This chapter studies emerging cyber-attacks on reinforcement learning (RL) We analyze the performance degradation of TD($lambda$) and $Q$-learning algorithms under the manipulation. A case study of TD($lambda$) learning is provided to corroborate the results.
• Score: 22.755411056179813
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This chapter studies emerging cyber-attacks on reinforcement learning (RL) and introduces a quantitative approach to analyze the vulnerabilities of RL. Focusing on adversarial manipulation on the cost signals, we analyze the performance degradation of TD($\lambda$) and $Q$-learning algorithms under the manipulation. For TD($\lambda$), the approximation learned from the manipulated costs has an approximation error bound proportional to the magnitude of the attack. The effect of the adversarial attacks on the bound does not depend on the choice of $\lambda$. In $Q$-learning, we show that $Q$-learning algorithms converge under stealthy attacks and bounded falsifications on cost signals. We characterize the relation between the falsified cost and the $Q$-factors as well as the policy learned by the learning agent which provides fundamental limits for feasible offensive and defensive moves. We propose a robust region in terms of the cost within which the adversary can never achieve the targeted policy. We provide conditions on the falsified cost which can mislead the agent to learn an adversary's favored policy. A case study of TD($\lambda$) learning is provided to corroborate the results.

Related papers

• Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning [49.48615590763914]
  We propose a black-box attack algorithm named LCBT, which uses the Monte Carlo tree search method for efficient action searching and manipulation. We conduct our proposed attack methods on three aggressive algorithms: DDPG, PPO, and TD3 in continuous settings, which show a promising attack performance.
  arXiv  Detail & Related papers  (2024-11-20T08:20:29Z)
• Adversarially Robust Deep Learning with Optimal-Transport-Regularized Divergences [12.1942837946862]
  We introduce the $ARMOR_D$ methods as novel approaches to enhancing the adversarial robustness of deep learning models. We demonstrate the effectiveness of our method on malware detection and image recognition applications.
  arXiv  Detail & Related papers  (2023-09-07T15:41:45Z)
• Near-Optimal Adversarial Reinforcement Learning with Switching Costs [43.895798638743784]
  We show how to develop a provably efficient algorithm for adversarial RL with switching costs. Our lower bound indicates that, due to the fundamental challenge of switching costs in adversarial RL, the best achieved regret is no longer achievable. We propose two novel switching-reduced algorithms with regrets that match our lower bound when the transition function is known.
  arXiv  Detail & Related papers  (2023-02-08T23:41:29Z)
• Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning [36.30086280732181]
  This paper studies poisoning attacks to manipulate emphany order-optimal learning algorithm towards a targeted policy in episodic RL. We find that the effect of attacks crucially depend on whether the rewards are bounded or unbounded.
  arXiv  Detail & Related papers  (2022-08-29T15:10:14Z)
• Reinforcement Learning for Linear Quadratic Control is Vulnerable Under Cost Manipulation [22.755411056179813]
  We study the deception of a Linear-Quadratic-Gaussian (LQG) agent by manipulating the cost signals. We show that a small falsification on the cost parameters will only lead to a bounded change in the optimal policy.
  arXiv  Detail & Related papers  (2022-03-11T06:59:42Z)
• Projective Ranking-based GNN Evasion Attacks [52.85890533994233]
  Graph neural networks (GNNs) offer promising learning methods for graph-related tasks. GNNs are at risk of adversarial attacks.
  arXiv  Detail & Related papers  (2022-02-25T21:52:09Z)
• Online Apprenticeship Learning [58.45089581278177]
  In Apprenticeship Learning (AL), we are given a Markov Decision Process (MDP) without access to the cost function. The goal is to find a policy that matches the expert's performance on some predefined set of cost functions. We show that the OAL problem can be effectively solved by combining two mirror descent based no-regret algorithms.
  arXiv  Detail & Related papers  (2021-02-13T12:57:51Z)
• Disturbing Reinforcement Learning Agents with Corrupted Rewards [62.997667081978825]
  We analyze the effects of different attack strategies based on reward perturbations on reinforcement learning algorithms. We show that smoothly crafting adversarial rewards are able to mislead the learner, and that using low exploration probability values, the policy learned is more robust to corrupt rewards.
  arXiv  Detail & Related papers  (2021-02-12T15:53:48Z)
• Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
  Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs. We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2020-08-05T07:49:42Z)
• Upper Confidence Primal-Dual Reinforcement Learning for CMDP with Adversarial Loss [145.54544979467872]
  We consider online learning for episodically constrained Markov decision processes (CMDPs) We propose a new emphupper confidence primal-dual algorithm, which only requires the trajectories sampled from the transition model. Our analysis incorporates a new high-probability drift analysis of Lagrange multiplier processes into the celebrated regret analysis of upper confidence reinforcement learning.
  arXiv  Detail & Related papers  (2020-03-02T05:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.