Policy Information Capacity: Information-Theoretic Measure for Task
Complexity in Deep Reinforcement Learning
- URL: http://arxiv.org/abs/2103.12726v1
- Date: Tue, 23 Mar 2021 17:49:50 GMT
- Title: Policy Information Capacity: Information-Theoretic Measure for Task
Complexity in Deep Reinforcement Learning
- Authors: Hiroki Furuta, Tatsuya Matsushima, Tadashi Kozuno, Yutaka Matsuo,
Sergey Levine, Ofir Nachum, Shixiang Shane Gu
- Abstract summary: We propose two environment-agnostic, algorithm-agnostic quantitative metrics for task difficulty.
We show that these metrics have higher correlations with normalized task solvability scores than a variety of alternatives.
These metrics can also be used for fast and compute-efficient optimizations of key design parameters.
- Score: 83.66080019570461
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Progress in deep reinforcement learning (RL) research is largely enabled by
benchmark task environments. However, analyzing the nature of those
environments is often overlooked. In particular, we still do not have agreeable
ways to measure the difficulty or solvability of a task, given that each has
fundamentally different actions, observations, dynamics, rewards, and can be
tackled with diverse RL algorithms. In this work, we propose policy information
capacity (PIC) -- the mutual information between policy parameters and episodic
return -- and policy-optimal information capacity (POIC) -- between policy
parameters and episodic optimality -- as two environment-agnostic,
algorithm-agnostic quantitative metrics for task difficulty. Evaluating our
metrics across toy environments as well as continuous control benchmark tasks
from OpenAI Gym and DeepMind Control Suite, we empirically demonstrate that
these information-theoretic metrics have higher correlations with normalized
task solvability scores than a variety of alternatives. Lastly, we show that
these metrics can also be used for fast and compute-efficient optimizations of
key design parameters such as reward shaping, policy architectures, and MDP
properties for better solvability by RL algorithms without ever running full RL
experiments.
Related papers
- Reparameterized Policy Learning for Multimodal Trajectory Optimization [61.13228961771765]
We investigate the challenge of parametrizing policies for reinforcement learning in high-dimensional continuous action spaces.
We propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories.
We present a practical model-based RL method, which leverages the multimodal policy parameterization and learned world model.
arXiv Detail & Related papers (2023-07-20T09:05:46Z) - Stepsize Learning for Policy Gradient Methods in Contextual Markov
Decision Processes [35.889129338603446]
Policy-based algorithms are among the most widely adopted techniques in model-free RL.
They tend to struggle when asked to accomplish a series of heterogeneous tasks.
We introduce a new formulation, known as meta-MDP, that can be used to solve any hyper parameter selection problem in RL.
arXiv Detail & Related papers (2023-06-13T12:58:12Z) - A Survey of Meta-Reinforcement Learning [83.95180398234238]
We cast the development of better RL algorithms as a machine learning problem itself in a process called meta-RL.
We discuss how, at a high level, meta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each individual task.
We conclude by presenting the open problems on the path to making meta-RL part of the standard toolbox for a deep RL practitioner.
arXiv Detail & Related papers (2023-01-19T12:01:41Z) - Hypernetworks for Zero-shot Transfer in Reinforcement Learning [21.994654567458017]
Hypernetworks are trained to generate behaviors across a range of unseen task conditions.
This work relates to meta RL, contextual RL, and transfer learning.
Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.
arXiv Detail & Related papers (2022-11-28T15:48:35Z) - Addressing the issue of stochastic environments and local
decision-making in multi-objective reinforcement learning [0.0]
Multi-objective reinforcement learning (MORL) is a relatively new field which builds on conventional Reinforcement Learning (RL)
This thesis focuses on what factors influence the frequency with which value-based MORL Q-learning algorithms learn the optimal policy for an environment.
arXiv Detail & Related papers (2022-11-16T04:56:42Z) - Exploration via Planning for Information about the Optimal Trajectory [67.33886176127578]
We develop a method that allows us to plan for exploration while taking the task and the current knowledge into account.
We demonstrate that our method learns strong policies with 2x fewer samples than strong exploration baselines.
arXiv Detail & Related papers (2022-10-06T20:28:55Z) - Neuroevolution is a Competitive Alternative to Reinforcement Learning
for Skill Discovery [12.586875201983778]
Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks.
We show that Quality Diversity (QD) methods are a competitive alternative to information-theory-augmented RL for skill discovery.
arXiv Detail & Related papers (2022-10-06T11:06:39Z) - Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients [54.98496284653234]
We consider the task of training a policy that maximizes reward while minimizing disclosure of certain sensitive state variables through the actions.
We solve this problem by introducing a regularizer based on the mutual information between the sensitive state and the actions.
We develop a model-based estimator for optimization of privacy-constrained policies.
arXiv Detail & Related papers (2020-12-30T03:22:35Z) - Learning Adaptive Exploration Strategies in Dynamic Environments Through
Informed Policy Regularization [100.72335252255989]
We study the problem of learning exploration-exploitation strategies that effectively adapt to dynamic environments.
We propose a novel algorithm that regularizes the training of an RNN-based policy using informed policies trained to maximize the reward in each task.
arXiv Detail & Related papers (2020-05-06T16:14:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.