Policy Entropy for Out-of-Distribution Classification
- URL: http://arxiv.org/abs/2005.12069v1
- Date: Mon, 25 May 2020 12:18:20 GMT
- Title: Policy Entropy for Out-of-Distribution Classification
- Authors: Andreas Sedlmeier and Robert M\"uller and Steffen Illium and Claudia
Linnhoff-Popien
- Abstract summary: We propose PEOC, a new policy entropy based out-of-distribution classifier.
It reliably detects unencountered states in deep reinforcement learning.
It is highly competitive against state-of-the-art one-class classification algorithms.
- Score: 8.747840760772268
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: One critical prerequisite for the deployment of reinforcement learning
systems in the real world is the ability to reliably detect situations on which
the agent was not trained. Such situations could lead to potential safety risks
when wrong predictions lead to the execution of harmful actions. In this work,
we propose PEOC, a new policy entropy based out-of-distribution classifier that
reliably detects unencountered states in deep reinforcement learning. It is
based on using the entropy of an agent's policy as the classification score of
a one-class classifier. We evaluate our approach using a procedural environment
generator. Results show that PEOC is highly competitive against
state-of-the-art one-class classification algorithms on the evaluated
environments. Furthermore, we present a structured process for benchmarking
out-of-distribution classification in reinforcement learning.
Related papers
- Hierarchical Selective Classification [17.136832159667204]
This paper introduces hierarchical selective classification, extending selective classification to a hierarchical setting.
We first formalize hierarchical risk and coverage, and introduce hierarchical risk-coverage curves.
Next, we develop algorithms for hierarchical selective classification, and propose an efficient algorithm that guarantees a target accuracy constraint with high probability.
arXiv Detail & Related papers (2024-05-19T12:24:30Z) - Interpretable Reinforcement Learning with Multilevel Subgoal Discovery [77.34726150561087]
We propose a novel Reinforcement Learning model for discrete environments.
In the model, an agent learns information about environment in the form of probabilistic rules.
No reward function is required for learning; an agent only needs to be given a primary goal to achieve.
arXiv Detail & Related papers (2022-02-15T14:04:44Z) - Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm.
Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
arXiv Detail & Related papers (2022-01-26T19:59:14Z) - Verified Probabilistic Policies for Deep Reinforcement Learning [6.85316573653194]
We tackle the problem of verifying probabilistic policies for deep reinforcement learning.
We propose an abstraction approach, based on interval Markov decision processes, that yields guarantees on a policy's execution.
We present techniques to build and solve these models using abstract interpretation, mixed-integer linear programming, entropy-based refinement and probabilistic model checking.
arXiv Detail & Related papers (2022-01-10T23:55:04Z) - Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs.
We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
arXiv Detail & Related papers (2021-06-21T21:42:08Z) - CASA-B: A Unified Framework of Model-Free Reinforcement Learning [1.4566990078034239]
CASA-B is an actor-critic framework that estimates state-value, state-action-value and policy.
We prove that CASA-B integrates a consistent path for the policy evaluation and the policy improvement.
We propose a progressive closed-form entropy control mechanism, which explicitly controls the behavior policies' entropy to arbitrary range.
arXiv Detail & Related papers (2021-05-09T12:45:13Z) - Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy.
We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
arXiv Detail & Related papers (2020-11-08T23:16:19Z) - Selective Classification via One-Sided Prediction [54.05407231648068]
One-sided prediction (OSP) based relaxation yields an SC scheme that attains near-optimal coverage in the practically relevant high target accuracy regime.
We theoretically derive bounds generalization for SC and OSP, and empirically we show that our scheme strongly outperforms state of the art methods in coverage at small error levels.
arXiv Detail & Related papers (2020-10-15T16:14:27Z) - Implicit Distributional Reinforcement Learning [61.166030238490634]
implicit distributional actor-critic (IDAC) built on two deep generator networks (DGNs)
Semi-implicit actor (SIA) powered by a flexible policy distribution.
We observe IDAC outperforms state-of-the-art algorithms on representative OpenAI Gym environments.
arXiv Detail & Related papers (2020-07-13T02:52:18Z) - In Automation We Trust: Investigating the Role of Uncertainty in Active
Learning Systems [5.459797813771497]
We investigate how different active learning (AL) query policies coupled with classification uncertainty visualizations affect analyst trust in automated classification systems.
We find that query policy significantly influences an analyst's trust in an image classification system.
We propose a set of oracle query policies and visualizations for use during AL training phases that can influence analyst trust in classification.
arXiv Detail & Related papers (2020-04-02T00:52:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.