A Definition of Continual Reinforcement Learning
- URL: http://arxiv.org/abs/2307.11046v2
- Date: Fri, 1 Dec 2023 13:52:32 GMT
- Title: A Definition of Continual Reinforcement Learning
- Authors: David Abel, Andr\'e Barreto, Benjamin Van Roy, Doina Precup, Hado van
Hasselt, Satinder Singh
- Abstract summary: In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward.
Continual reinforcement learning refers to the setting in which the best agents never stop learning.
We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents.
- Score: 69.56273766737527
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In a standard view of the reinforcement learning problem, an agent's goal is
to efficiently identify a policy that maximizes long-term reward. However, this
perspective is based on a restricted view of learning as finding a solution,
rather than treating learning as endless adaptation. In contrast, continual
reinforcement learning refers to the setting in which the best agents never
stop learning. Despite the importance of continual reinforcement learning, the
community lacks a simple definition of the problem that highlights its
commitments and makes its primary concepts precise and clear. To this end, this
paper is dedicated to carefully defining the continual reinforcement learning
problem. We formalize the notion of agents that "never stop learning" through a
new mathematical language for analyzing and cataloging agents. Using this new
language, we define a continual learning agent as one that can be understood as
carrying out an implicit search process indefinitely, and continual
reinforcement learning as the setting in which the best agents are all
continual learning agents. We provide two motivating examples, illustrating
that traditional views of multi-task reinforcement learning and continual
supervised learning are special cases of our definition. Collectively, these
definitions and perspectives formalize many intuitive concepts at the heart of
learning, and open new research pathways surrounding continual learning agents.
Related papers
- A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents [18.2920082469313]
We argue that open-ended learning is generally conceived as a composite notion encompassing a set of diverse properties.
We focus on the subset of open-ended goal-conditioned reinforcement learning problems in which agents can learn a growing repertoire of goal-driven skills.
arXiv Detail & Related papers (2023-11-01T07:37:27Z) - Causal Reinforcement Learning: A Survey [57.368108154871]
Reinforcement learning is an essential paradigm for solving sequential decision problems under uncertainty.
One of the main obstacles is that reinforcement learning agents lack a fundamental understanding of the world.
Causality offers a notable advantage as it can formalize knowledge in a systematic manner.
arXiv Detail & Related papers (2023-07-04T03:00:43Z) - A Comprehensive Survey of Continual Learning: Theory, Method and
Application [64.23253420555989]
We present a comprehensive survey of continual learning, seeking to bridge the basic settings, theoretical foundations, representative methods, and practical applications.
We summarize the general objectives of continual learning as ensuring a proper stability-plasticity trade-off and an adequate intra/inter-task generalizability in the context of resource efficiency.
arXiv Detail & Related papers (2023-01-31T11:34:56Z) - Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learning [18.61040106667249]
In AI, generalization refers to a model's ability to perform well on out-of-distribution data related to a given task, beyond the data it was trained on.
Continual learning methods often include mechanisms to mitigate catastrophic forgetting, ensuring that knowledge from earlier tasks is retained.
We introduce a simple and effective technique known as Shape-Texture Consistency Regularization (STCR), which caters to continual learning.
arXiv Detail & Related papers (2022-11-21T04:36:24Z) - A Review of Uncertainty for Deep Reinforcement Learning [0.0]
Uncertainty is ubiquitous in games, both in the agents playing games and often in the games themselves.
This work provides an overview motivating and presenting existing techniques in uncertainty aware deep reinforcement learning.
arXiv Detail & Related papers (2022-08-18T20:42:19Z) - Rethinking Learning Dynamics in RL using Adversarial Networks [79.56118674435844]
We present a learning mechanism for reinforcement learning of closely related skills parameterized via a skill embedding space.
The main contribution of our work is to formulate an adversarial training regime for reinforcement learning with the help of entropy-regularized policy gradient formulation.
arXiv Detail & Related papers (2022-01-27T19:51:09Z) - Towards a theory of out-of-distribution learning [23.878004729029644]
We propose a chronological approach to defining different learning tasks using the provably approximately correct (PAC) learning framework.
We will start with in-distribution learning and progress to recently proposed lifelong or continual learning.
Our hope is that this work will inspire a universally agreed-upon approach to quantifying different types of learning.
arXiv Detail & Related papers (2021-09-29T15:35:16Z) - Importance Weighted Policy Learning and Adaptation [89.46467771037054]
We study a complementary approach which is conceptually simple, general, modular and built on top of recent improvements in off-policy learning.
The framework is inspired by ideas from the probabilistic inference literature and combines robust off-policy learning with a behavior prior.
Our approach achieves competitive adaptation performance on hold-out tasks compared to meta reinforcement learning baselines and can scale to complex sparse-reward scenarios.
arXiv Detail & Related papers (2020-09-10T14:16:58Z) - Online Fast Adaptation and Knowledge Accumulation: a New Approach to
Continual Learning [74.07455280246212]
Continual learning studies agents that learn from streams of tasks without forgetting previous ones while adapting to new ones.
We show that current continual learning, meta-learning, meta-continual learning, and continual-meta learning techniques fail in this new scenario.
We propose Continual-MAML, an online extension of the popular MAML algorithm as a strong baseline for this scenario.
arXiv Detail & Related papers (2020-03-12T15:47:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.