Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness

• URL: http://arxiv.org/abs/2301.07487v1
• Date: Tue, 17 Jan 2023 16:54:33 GMT
• Title: Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness
• Authors: Ezgi Korkmaz
• Abstract summary: We show that high sensitivity directions are more abundant in the deep neural policy landscape and can be found via more natural means in a black-box setting. We show that vanilla training techniques intriguingly result in learning more robust policies compared to the policies learnt via the state-of-the-art adversarial training techniques.
• Score: 7.6146285961466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning from raw high dimensional data via interaction with a given environment has been effectively achieved through the utilization of deep neural networks. Yet the observed degradation in policy performance caused by imperceptible worst-case policy dependent translations along high sensitivity directions (i.e. adversarial perturbations) raises concerns on the robustness of deep reinforcement learning policies. In our paper, we show that these high sensitivity directions do not lie only along particular worst-case directions, but rather are more abundant in the deep neural policy landscape and can be found via more natural means in a black-box setting. Furthermore, we show that vanilla training techniques intriguingly result in learning more robust policies compared to the policies learnt via the state-of-the-art adversarial training techniques. We believe our work lays out intriguing properties of the deep reinforcement learning policy manifold and our results can help to build robust and generalizable deep reinforcement learning policies.

Related papers

• Understanding and Diagnosing Deep Reinforcement Learning [14.141453107129403]
  Deep neural policies have recently been installed in a diverse range of settings, from biotechnology to automated financial systems. We introduce a theoretically founded technique that provides a systematic analysis of the directions in the deep neural policy decision decision both time and space.
  arXiv  Detail & Related papers  (2024-06-23T18:10:16Z)
• Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions [8.173034693197351]
  We propose a novel method to detect the presence of non-robust directions in MDPs. Our method provides a theoretical basis for the fundamental cut-off between safe observations and adversarial observations. Most significantly, we demonstrate the effectiveness of our approach even in the setting where non-robust directions are explicitly optimized to circumvent our proposed method.
  arXiv  Detail & Related papers  (2023-06-09T13:11:05Z)
• Representation-Driven Reinforcement Learning [57.44609759155611]
  We present a representation-driven framework for reinforcement learning. By representing policies as estimates of their expected values, we leverage techniques from contextual bandits to guide exploration and exploitation. We demonstrate the effectiveness of this framework through its application to evolutionary and policy gradient-based approaches.
  arXiv  Detail & Related papers  (2023-05-31T14:59:12Z)
• Bounded Robustness in Reinforcement Learning via Lexicographic Objectives [54.00072722686121]
  Policy robustness in Reinforcement Learning may not be desirable at any cost. We study how policies can be maximally robust to arbitrary observational noise. We propose a robustness-inducing scheme, applicable to any policy algorithm, that trades off expected policy utility for robustness.
  arXiv  Detail & Related papers  (2022-09-30T08:53:18Z)
• Rethinking Learning Dynamics in RL using Adversarial Networks [79.56118674435844]
  We present a learning mechanism for reinforcement learning of closely related skills parameterized via a skill embedding space. The main contribution of our work is to formulate an adversarial training regime for reinforcement learning with the help of entropy-regularized policy gradient formulation.
  arXiv  Detail & Related papers  (2022-01-27T19:51:09Z)
• Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs [0.0]
  We propose a framework to investigate the decision boundary and loss landscape similarities across states and across MDPs. We conduct experiments in various games from Arcade Learning Environment, and discover that high sensitivity directions for neural policies are correlated across MDPs.
  arXiv  Detail & Related papers  (2021-12-16T17:10:41Z)
• Where Did You Learn That From? Surprising Effectiveness of Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learning [114.9857000195174]
  A major challenge to widespread industrial adoption of deep reinforcement learning is the potential vulnerability to privacy breaches. We propose an adversarial attack framework tailored for testing the vulnerability of deep reinforcement learning algorithms to membership inference attacks.
  arXiv  Detail & Related papers  (2021-09-08T23:44:57Z)
• Investigating Vulnerabilities of Deep Neural Policies [0.0]
  Reinforcement learning policies based on deep neural networks are vulnerable to imperceptible adversarial perturbations to their inputs. Recent work has proposed several methods to improve the robustness of deep reinforcement learning agents to adversarial perturbations. We study the effects of adversarial training on the neural policy learned by the agent.
  arXiv  Detail & Related papers  (2021-08-30T10:04:50Z)
• Improving Robustness of Deep Reinforcement Learning Agents: Environment Attacks based on Critic Networks [12.521494095948068]
  A line of recent works focus on producing disturbances of the environment. Existing approaches of the literature to generate meaningful disturbances of the environment are adversarial reinforcement learning methods. We show that our method, while being faster and lighter, leads to significantly better improvements in policy robustness than existing methods of the literature.
  arXiv  Detail & Related papers  (2021-04-07T14:37:23Z)
• Optimism in the Face of Adversity: Understanding and Improving Deep Learning through Adversarial Robustness [63.627760598441796]
  We provide an in-depth review of the field of adversarial robustness in deep learning. We highlight the intuitive connection between adversarial examples and the geometry of deep neural networks. We provide an overview of the main emerging applications of adversarial robustness beyond security.
  arXiv  Detail & Related papers  (2020-10-19T16:03:46Z)
• Reward-Conditioned Policies [100.64167842905069]
  imitation learning requires near-optimal expert data. Can we learn effective policies via supervised learning without demonstrations? We show how such an approach can be derived as a principled method for policy search.
  arXiv  Detail & Related papers  (2019-12-31T18:07:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.