Policy Entropy for Out-of-Distribution Classification

• URL: http://arxiv.org/abs/2005.12069v1
• Date: Mon, 25 May 2020 12:18:20 GMT
• Title: Policy Entropy for Out-of-Distribution Classification
• Authors: Andreas Sedlmeier and Robert M\"uller and Steffen Illium and Claudia Linnhoff-Popien
• Abstract summary: We propose PEOC, a new policy entropy based out-of-distribution classifier. It reliably detects unencountered states in deep reinforcement learning. It is highly competitive against state-of-the-art one-class classification algorithms.
• Score: 8.747840760772268
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One critical prerequisite for the deployment of reinforcement learning systems in the real world is the ability to reliably detect situations on which the agent was not trained. Such situations could lead to potential safety risks when wrong predictions lead to the execution of harmful actions. In this work, we propose PEOC, a new policy entropy based out-of-distribution classifier that reliably detects unencountered states in deep reinforcement learning. It is based on using the entropy of an agent's policy as the classification score of a one-class classifier. We evaluate our approach using a procedural environment generator. Results show that PEOC is highly competitive against state-of-the-art one-class classification algorithms on the evaluated environments. Furthermore, we present a structured process for benchmarking out-of-distribution classification in reinforcement learning.

Related papers

• Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank [64.44255178199846]
  We propose a novel approach, proximal ranking policy optimization (PRPO), that provides safety in deployment without assumptions about user behavior. PRPO removes incentives for learning ranking behavior that is too dissimilar to a safe ranking model. Our experiments show that PRPO provides higher performance than the existing safe inverse propensity scoring approach.
  arXiv  Detail & Related papers  (2024-09-15T22:22:27Z)
• Hierarchical Selective Classification [17.136832159667204]
  This paper introduces hierarchical selective classification, extending selective classification to a hierarchical setting. We first formalize hierarchical risk and coverage, and introduce hierarchical risk-coverage curves. Next, we develop algorithms for hierarchical selective classification, and propose an efficient algorithm that guarantees a target accuracy constraint with high probability.
  arXiv  Detail & Related papers  (2024-05-19T12:24:30Z)
• PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm [28.48626438603237]
  PACER consists of a distributional critic, an actor and a sample-based encourager. Push-forward operator is leveraged in both the critic and actor to model the return distributions and policies respectively. A sample-based utility value policy gradient is established for the push-forward policy update.
  arXiv  Detail & Related papers  (2023-06-11T09:45:31Z)
• Interpretable Reinforcement Learning with Multilevel Subgoal Discovery [77.34726150561087]
  We propose a novel Reinforcement Learning model for discrete environments. In the model, an agent learns information about environment in the form of probabilistic rules. No reward function is required for learning; an agent only needs to be given a primary goal to achieve.
  arXiv  Detail & Related papers  (2022-02-15T14:04:44Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• Verified Probabilistic Policies for Deep Reinforcement Learning [6.85316573653194]
  We tackle the problem of verifying probabilistic policies for deep reinforcement learning. We propose an abstraction approach, based on interval Markov decision processes, that yields guarantees on a policy's execution. We present techniques to build and solve these models using abstract interpretation, mixed-integer linear programming, entropy-based refinement and probabilistic model checking.
  arXiv  Detail & Related papers  (2022-01-10T23:55:04Z)
• CASA-B: A Unified Framework of Model-Free Reinforcement Learning [1.4566990078034239]
  CASA-B is an actor-critic framework that estimates state-value, state-action-value and policy. We prove that CASA-B integrates a consistent path for the policy evaluation and the policy improvement. We propose a progressive closed-form entropy control mechanism, which explicitly controls the behavior policies' entropy to arbitrary range.
  arXiv  Detail & Related papers  (2021-05-09T12:45:13Z)
• Reliable Off-policy Evaluation for Reinforcement Learning [53.486680020852724]
  In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy. We propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged data.
  arXiv  Detail & Related papers  (2020-11-08T23:16:19Z)
• Selective Classification via One-Sided Prediction [54.05407231648068]
  One-sided prediction (OSP) based relaxation yields an SC scheme that attains near-optimal coverage in the practically relevant high target accuracy regime. We theoretically derive bounds generalization for SC and OSP, and empirically we show that our scheme strongly outperforms state of the art methods in coverage at small error levels.
  arXiv  Detail & Related papers  (2020-10-15T16:14:27Z)
• Implicit Distributional Reinforcement Learning [61.166030238490634]
  implicit distributional actor-critic (IDAC) built on two deep generator networks (DGNs) Semi-implicit actor (SIA) powered by a flexible policy distribution. We observe IDAC outperforms state-of-the-art algorithms on representative OpenAI Gym environments.
  arXiv  Detail & Related papers  (2020-07-13T02:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.