Optimal Attack and Defense for Reinforcement Learning

• URL: http://arxiv.org/abs/2312.00198v2
• Date: Mon, 17 Jun 2024 08:13:44 GMT
• Title: Optimal Attack and Defense for Reinforcement Learning
• Authors: Jeremy McMahan, Young Wu, Xiaojin Zhu, Qiaomin Xie,
• Abstract summary: In adversarial RL, an external attacker has the power to manipulate the victim agent's interaction with the environment. We show the attacker's problem of designing a stealthy attack that maximizes its own expected reward. We argue that the optimal defense policy for the victim can be computed as the solution to a Stackelberg game.
• Score: 11.36770403327493
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To ensure the usefulness of Reinforcement Learning (RL) in real systems, it is crucial to ensure they are robust to noise and adversarial attacks. In adversarial RL, an external attacker has the power to manipulate the victim agent's interaction with the environment. We study the full class of online manipulation attacks, which include (i) state attacks, (ii) observation attacks (which are a generalization of perceived-state attacks), (iii) action attacks, and (iv) reward attacks. We show the attacker's problem of designing a stealthy attack that maximizes its own expected reward, which often corresponds to minimizing the victim's value, is captured by a Markov Decision Process (MDP) that we call a meta-MDP since it is not the true environment but a higher level environment induced by the attacked interaction. We show that the attacker can derive optimal attacks by planning in polynomial time or learning with polynomial sample complexity using standard RL techniques. We argue that the optimal defense policy for the victim can be computed as the solution to a stochastic Stackelberg game, which can be further simplified into a partially-observable turn-based stochastic game (POTBSG). Neither the attacker nor the victim would benefit from deviating from their respective optimal policies, thus such solutions are truly robust. Although the defense problem is NP-hard, we show that optimal Markovian defenses can be computed (learned) in polynomial time (sample complexity) in many scenarios.

Related papers

• Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attacks [62.036798488144306]
  Current defense mainly focuses on the known attacks, but the adversarial robustness to the unknown attacks is seriously overlooked. We propose an attack-agnostic defense method named Meta Invariance Defense (MID) We show that MID simultaneously achieves robustness to the imperceptible adversarial perturbations in high-level image classification and attack-suppression in low-level robust image regeneration.
  arXiv  Detail & Related papers  (2024-04-04T10:10:38Z)
• Securing Recommender System via Cooperative Training [78.97620275467733]
  We propose a general framework, Triple Cooperative Defense (TCD), which employs three cooperative models that mutually enhance data. Considering existing attacks struggle to balance bi-level optimization and efficiency, we revisit poisoning attacks in recommender systems. We put forth a Game-based Co-training Attack (GCoAttack), which frames the proposed CoAttack and TCD as a game-theoretic process.
  arXiv  Detail & Related papers  (2024-01-23T12:07:20Z)
• Optimal Cost Constrained Adversarial Attacks For Multiple Agent Systems [6.69087470775851]
  We formulate the problem of performing optimal adversarial agent-to-agent attacks using distributed attack agents. We propose an optimal method integrating within-step static constrained attack-resource allocation optimization and between-step dynamic programming. Our numerical results show that the proposed attacks can significantly reduce the rewards received by the attacked agents.
  arXiv  Detail & Related papers  (2023-11-01T21:28:02Z)
• Planning for Attacker Entrapment in Adversarial Settings [16.085007590604327]
  We propose a framework to generate a defense strategy against an attacker who is working in an environment where a defender can operate without the attacker's knowledge. Our problem formulation allows us to capture it as a much simpler infinite horizon discounted MDP, in which the optimal policy for the MDP gives the defender's strategy against the actions of the attacker.
  arXiv  Detail & Related papers  (2023-03-01T21:08:27Z)
• Guidance Through Surrogate: Towards a Generic Diagnostic Attack [101.36906370355435]
  We develop a guided mechanism to avoid local minima during attack optimization, leading to a novel attack dubbed Guided Projected Gradient Attack (G-PGA) Our modified attack does not require random restarts, large number of attack iterations or search for an optimal step-size. More than an effective attack, G-PGA can be used as a diagnostic tool to reveal elusive robustness due to gradient masking in adversarial defenses.
  arXiv  Detail & Related papers  (2022-12-30T18:45:23Z)
• Sampling Attacks on Meta Reinforcement Learning: A Minimax Formulation and Complexity Analysis [20.11993437283895]
  This paper provides a game-theoretical underpinning for understanding this type of security risk. We define the sampling attack model as a Stackelberg game between the attacker and the agent, which yields a minimax formulation. We observe that a minor effort of the attacker can significantly deteriorate the learning performance.
  arXiv  Detail & Related papers  (2022-07-29T21:29:29Z)
• Attacking and Defending Deep Reinforcement Learning Policies [3.6985039575807246]
  We study robustness of DRL policies to adversarial attacks from the perspective of robust optimization. We propose a greedy attack algorithm, which tries to minimize the expected return of the policy without interacting with the environment, and a defense algorithm, which performs adversarial training in a max-min form.
  arXiv  Detail & Related papers  (2022-05-16T12:47:54Z)
• Projective Ranking-based GNN Evasion Attacks [52.85890533994233]
  Graph neural networks (GNNs) offer promising learning methods for graph-related tasks. GNNs are at risk of adversarial attacks.
  arXiv  Detail & Related papers  (2022-02-25T21:52:09Z)
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS [70.60975663021952]
  We study blackbox adversarial attacks on network classifiers. We argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions. We show that a continual learning approach is required to study attacker-defender dynamics.
  arXiv  Detail & Related papers  (2021-11-23T23:42:16Z)
• Composite Adversarial Attacks [57.293211764569996]
  Adversarial attack is a technique for deceiving Machine Learning (ML) models. In this paper, a new procedure called Composite Adrial Attack (CAA) is proposed for automatically searching the best combination of attack algorithms. CAA beats 10 top attackers on 11 diverse defenses with less elapsed time.
  arXiv  Detail & Related papers  (2020-12-10T03:21:16Z)
• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning [33.41280432984183]
  We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy. As a victim, we consider RL agents whose objective is to find a policy that maximizes average reward in undiscounted infinite-horizon problem settings.
  arXiv  Detail & Related papers  (2020-03-28T23:22:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.