Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions

• URL: http://arxiv.org/abs/2306.05873v1
• Date: Fri, 9 Jun 2023 13:11:05 GMT
• Title: Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions
• Authors: Ezgi Korkmaz, Jonah Brown-Cohen
• Abstract summary: We propose a novel method to detect the presence of non-robust directions in MDPs. Our method provides a theoretical basis for the fundamental cut-off between safe observations and adversarial observations. Most significantly, we demonstrate the effectiveness of our approach even in the setting where non-robust directions are explicitly optimized to circumvent our proposed method.
• Score: 8.173034693197351
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning in MDPs with highly complex state representations is currently possible due to multiple advancements in reinforcement learning algorithm design. However, this incline in complexity, and furthermore the increase in the dimensions of the observation came at the cost of volatility that can be taken advantage of via adversarial attacks (i.e. moving along worst-case directions in the observation space). To solve this policy instability problem we propose a novel method to detect the presence of these non-robust directions via local quadratic approximation of the deep neural policy loss. Our method provides a theoretical basis for the fundamental cut-off between safe observations and adversarial observations. Furthermore, our technique is computationally efficient, and does not depend on the methods used to produce the worst-case directions. We conduct extensive experiments in the Arcade Learning Environment with several different adversarial attack techniques. Most significantly, we demonstrate the effectiveness of our approach even in the setting where non-robust directions are explicitly optimized to circumvent our proposed method.

Related papers

• Understanding and Diagnosing Deep Reinforcement Learning [14.141453107129403]
  Deep neural policies have recently been installed in a diverse range of settings, from biotechnology to automated financial systems. We introduce a theoretically founded technique that provides a systematic analysis of the directions in the deep neural policy decision decision both time and space.
  arXiv  Detail & Related papers  (2024-06-23T18:10:16Z)
• READ: Improving Relation Extraction from an ADversarial Perspective [33.44949503459933]
  We propose an adversarial training method specifically designed for relation extraction (RE) Our approach introduces both sequence- and token-level perturbations to the sample and uses a separate perturbation vocabulary to improve the search for entity and context perturbations.
  arXiv  Detail & Related papers  (2024-04-02T16:42:44Z)
• Adaptive trajectory-constrained exploration strategy for deep reinforcement learning [6.589742080994319]
  Deep reinforcement learning (DRL) faces significant challenges in addressing the hard-exploration problems in tasks with sparse or deceptive rewards and large state spaces. We propose an efficient adaptive trajectory-constrained exploration strategy for DRL. We conduct experiments on two large 2D grid world mazes and several MuJoCo tasks.
  arXiv  Detail & Related papers  (2023-12-27T07:57:15Z)
• Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics [96.9177297872723]
  We present a novel method for guaranteeing linear momentum in learned physics simulations. We enforce conservation of momentum with a hard constraint, which we realize via antisymmetrical continuous convolutional layers. In combination, the proposed method allows us to increase the physical accuracy of the learned simulator substantially.
  arXiv  Detail & Related papers  (2022-10-12T09:12:59Z)
• Improving robustness of jet tagging algorithms with adversarial training [56.79800815519762]
  We investigate the vulnerability of flavor tagging algorithms via application of adversarial attacks. We present an adversarial training strategy that mitigates the impact of such simulated attacks.
  arXiv  Detail & Related papers  (2022-03-25T19:57:19Z)
• Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs [0.0]
  We propose a framework to investigate the decision boundary and loss landscape similarities across states and across MDPs. We conduct experiments in various games from Arcade Learning Environment, and discover that high sensitivity directions for neural policies are correlated across MDPs.
  arXiv  Detail & Related papers  (2021-12-16T17:10:41Z)
• Surveillance Evasion Through Bayesian Reinforcement Learning [78.79938727251594]
  We consider a 2D continuous path planning problem with a completely unknown intensity of random termination. Those Observers' surveillance intensity is a priori unknown and has to be learned through repetitive path planning.
  arXiv  Detail & Related papers  (2021-09-30T02:29:21Z)
• A Hamiltonian Monte Carlo Method for Probabilistic Adversarial Attack and Learning [122.49765136434353]
  We present an effective method, called Hamiltonian Monte Carlo with Accumulated Momentum (HMCAM), aiming to generate a sequence of adversarial examples. We also propose a new generative method called Contrastive Adversarial Training (CAT), which approaches equilibrium distribution of adversarial examples. Both quantitative and qualitative analysis on several natural image datasets and practical systems have confirmed the superiority of the proposed algorithm.
  arXiv  Detail & Related papers  (2020-10-15T16:07:26Z)
• Efficient Exploration in Constrained Environments with Goal-Oriented Reference Path [15.679210057474922]
  We train a deep convolutional network that can predict collision-free paths based on a map of the environment. This is then used by a reinforcement learning algorithm to learn to closely follow the path. We show that our method consistently improves the sample efficiency and generalization capability to novel environments.
  arXiv  Detail & Related papers  (2020-03-03T17:07:47Z)
• Disentangling Adaptive Gradient Methods from Learning Rates [65.0397050979662]
  We take a deeper look at how adaptive gradient methods interact with the learning rate schedule. We introduce a "grafting" experiment which decouples an update's magnitude from its direction. We present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods.
  arXiv  Detail & Related papers  (2020-02-26T21:42:49Z)
• Discrete Action On-Policy Learning with Action-Value Critic [72.20609919995086]
  Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension. We construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques.
  arXiv  Detail & Related papers  (2020-02-10T04:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.