Probabilistic Perspectives on Error Minimization in Adversarial Reinforcement Learning

• URL: http://arxiv.org/abs/2406.04724v2
• Date: Sun, 06 Oct 2024 14:00:14 GMT
• Title: Probabilistic Perspectives on Error Minimization in Adversarial Reinforcement Learning
• Authors: Roman Belaire, Arunesh Sinha, Pradeep Varakantham,
• Abstract summary: A self-driving car could experience catastrophic consequences if sensory inputs about traffic signs are manipulated by an adversary. The core challenge in such situations is that the true state of the environment becomes only partially observable due to these adversarial manipulations. We introduce a novel objective called Adversarial Counterfactual Error (ACoE) which is defined on the beliefs about the underlying true state.
• Score: 18.044879441434432
• License:
• Abstract: Deep Reinforcement Learning (DRL) policies are highly susceptible to adversarial noise in observations, which poses significant risks in safety-critical scenarios. For instance, a self-driving car could experience catastrophic consequences if its sensory inputs about traffic signs are manipulated by an adversary. The core challenge in such situations is that the true state of the environment becomes only partially observable due to these adversarial manipulations. Two key strategies have so far been employed in the literature; the first set of methods focuses on increasing the likelihood that nearby states--those close to the true state--share the same robust actions. The second set of approaches maximize the value for the worst possible true state within the range of adversarially perturbed observations. Although these approaches provide strong robustness against attacks, they tend to be either overly conservative or not generalizable. We hypothesize that the shortcomings of these approaches stem from their failure to explicitly account for partial observability. By making decisions that directly consider this partial knowledge of the true state, we believe it is possible to achieve a better balance between robustness and performance, particularly in adversarial settings. To achieve this, we introduce a novel objective called Adversarial Counterfactual Error (ACoE), which is defined on the beliefs about the underlying true state and naturally balances value optimization with robustness against adversarial attacks, and a theoretically-grounded, scalable surrogate objective Cumulative-ACoE (C-ACoE). Our empirical evaluations demonstrate that our method significantly outperforms current state-of-the-art approaches for addressing adversarial RL challenges, offering a promising direction for better DRL under adversarial conditions.

Related papers

• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users. We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions. We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversariality [13.240598841087841]
  We introduce FREA, a novel safety-critical scenarios generation method that incorporates the Largest Feasible Region (LFR) of AV as guidance. Experiments illustrate that FREA can effectively generate safety-critical scenarios, yielding considerable near-miss events.
  arXiv  Detail & Related papers  (2024-06-05T06:26:15Z)
• Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations [5.076419064097735]
  Recent work shows that a well-trained RL agent can be easily manipulated by strategically perturbing its state observations at the test stage. Existing solutions either introduce a regularization term to improve the smoothness of the trained policy against perturbations or alternatively train the agent's policy and the attacker's policy. We propose a new robust RL algorithm for deriving a pessimistic policy to safeguard against an agent's uncertainty about true states.
  arXiv  Detail & Related papers  (2024-03-06T20:52:49Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Regret-Based Defense in Adversarial Reinforcement Learning [14.671837627588294]
  adversarial noise can have disastrous consequences in safety-critical environments. Existing approaches for making RL algorithms robust to an observation-perturbing adversary have focused on reactive approaches. We provide a principled approach that minimizes maximum regret over a "neighborhood" of observations to the received "observation"
  arXiv  Detail & Related papers  (2023-02-14T08:56:50Z)
• Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk [30.229387511344456]
  We propose a novel reinforcement learning algorithm of CVaR-Proximal-Policy-Optimization (CPPO) which formalizes the risk-sensitive constrained optimization problem by keeping its CVaR under a given threshold. Experimental results show that CPPO achieves a higher cumulative reward and is more robust against both observation and transition disturbances.
  arXiv  Detail & Related papers  (2022-06-09T11:57:54Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)
• Adversarial Visual Robustness by Causal Intervention [56.766342028800445]
  Adversarial training is the de facto most promising defense against adversarial examples. Yet, its passive nature inevitably prevents it from being immune to unknown attackers. We provide a causal viewpoint of adversarial vulnerability: the cause is the confounder ubiquitously existing in learning.
  arXiv  Detail & Related papers  (2021-06-17T14:23:54Z)
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary [86.0846119254031]
  We study the robustness of reinforcement learning with adversarially perturbed state observations. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
  arXiv  Detail & Related papers  (2021-01-21T05:38:52Z)
• Adversary Agnostic Robust Deep Reinforcement Learning [23.9114110755044]
  Deep reinforcement learning policies are deceived by perturbations during training. Previous approaches assume that the knowledge of adversaries can be added into the training process. We propose an adversary robust DRL paradigm that does not require learning from adversaries.
  arXiv  Detail & Related papers  (2020-08-14T06:04:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.