A Simple Unified Framework for Anomaly Detection in Deep Reinforcement
Learning
- URL: http://arxiv.org/abs/2109.09889v1
- Date: Tue, 21 Sep 2021 00:09:03 GMT
- Title: A Simple Unified Framework for Anomaly Detection in Deep Reinforcement
Learning
- Authors: Hongming Zhang, Ke Sun, Bo Xu, Linglong Kong, Martin M\"uller
- Abstract summary: Abnormal states in deep reinforcement learning(RL) are states that are beyond the scope of an RL policy.
We propose a simple yet effective anomaly detection framework for deep RL algorithms.
- Score: 20.08390854681034
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Abnormal states in deep reinforcement learning~(RL) are states that are
beyond the scope of an RL policy. Such states may make the RL system unsafe and
impede its deployment in real scenarios. In this paper, we propose a simple yet
effective anomaly detection framework for deep RL algorithms that
simultaneously considers random, adversarial and out-of-distribution~(OOD)
state outliers. In particular, we attain the class-conditional distributions
for each action class under the Gaussian assumption, and rely on these
distributions to discriminate between inliers and outliers based on Mahalanobis
Distance~(MD) and Robust Mahalanobis Distance. We conduct extensive experiments
on Atari games that verify the effectiveness of our detection strategies. To
the best of our knowledge, we present the first in-detail study of statistical
and adversarial anomaly detection in deep RL algorithms. This simple unified
anomaly detection paves the way towards deploying safe RL systems in real-world
applications.
Related papers
- Leveraging Reward Consistency for Interpretable Feature Discovery in
Reinforcement Learning [69.19840497497503]
It is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents.
We propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents.
We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment.
arXiv Detail & Related papers (2023-09-04T09:09:54Z) - Approximate Model-Based Shielding for Safe Reinforcement Learning [83.55437924143615]
We propose a principled look-ahead shielding algorithm for verifying the performance of learned RL policies.
Our algorithm differs from other shielding approaches in that it does not require prior knowledge of the safety-relevant dynamics of the system.
We demonstrate superior performance to other safety-aware approaches on a set of Atari games with state-dependent safety-labels.
arXiv Detail & Related papers (2023-07-27T15:19:45Z) - One-Step Distributional Reinforcement Learning [10.64435582017292]
We present the simpler one-step distributional reinforcement learning (OS-DistrRL) framework.
We show that our approach comes with a unified theory for both policy evaluation and control.
We propose two OS-DistrRL algorithms for which we provide an almost sure convergence analysis.
arXiv Detail & Related papers (2023-04-27T06:57:00Z) - Efficient Deep Reinforcement Learning Requires Regulating Overfitting [91.88004732618381]
We show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms.
We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
arXiv Detail & Related papers (2023-04-20T17:11:05Z) - Sample Efficient Deep Reinforcement Learning via Local Planning [21.420851589712626]
This work focuses on sample-efficient deep reinforcement learning (RL) with a simulator.
We propose an algorithmic framework, named uncertainty-first local planning (UFLP), that takes advantage of this property.
We demonstrate that this simple procedure can dramatically improve the sample cost of several baseline RL algorithms on difficult exploration tasks.
arXiv Detail & Related papers (2023-01-29T23:17:26Z) - Towards Open Set Video Anomaly Detection [11.944167192592905]
Open Set Video Anomaly Detection (OpenVAD) aims to identify abnormal events from video data where both known anomalies and novel ones exist in testing.
We develop a novel weakly supervised method for the OpenVAD problem by integrating evidential deep learning (EDL) and normalizing flows (NFs) into a multiple instance learning (MIL) framework.
arXiv Detail & Related papers (2022-08-23T17:53:34Z) - Anomaly Rule Detection in Sequence Data [2.3757190901941736]
We present a new anomaly detection framework called DUOS that enables Discovery of Utility-aware Outlier Sequential rules from a set of sequences.
In this work, we incorporate both the anomalousness and utility of a group, and then introduce the concept of utility-aware outlier rule (UOSR)
arXiv Detail & Related papers (2021-11-29T23:52:31Z) - GalilAI: Out-of-Task Distribution Detection using Causal Active
Experimentation for Safe Transfer RL [11.058960131490903]
Out-of-distribution (OOD) detection is a well-studied topic in supervised learning.
We propose a novel task: that of Out-of-Task Distribution (OOTD) detection.
We name our method GalilAI, in honor of Galileo Galilei, as it discovers, among other causal processes, that gravitational acceleration is independent of the mass of a body.
arXiv Detail & Related papers (2021-10-29T01:45:56Z) - Reannealing of Decaying Exploration Based On Heuristic Measure in Deep
Q-Network [82.20059754270302]
We propose an algorithm based on the idea of reannealing, that aims at encouraging exploration only when it is needed.
We perform an illustrative case study showing that it has potential to both accelerate training and obtain a better policy.
arXiv Detail & Related papers (2020-09-29T20:40:00Z) - Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs.
We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
arXiv Detail & Related papers (2020-08-05T07:49:42Z) - Robust Deep Reinforcement Learning against Adversarial Perturbations on
State Observations [88.94162416324505]
A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises.
Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions.
We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
arXiv Detail & Related papers (2020-03-19T17:59:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.