Meta Automatic Curriculum Learning
- URL: http://arxiv.org/abs/2011.08463v3
- Date: Wed, 1 Sep 2021 15:41:34 GMT
- Title: Meta Automatic Curriculum Learning
- Authors: R\'emy Portelas, Cl\'ement Romac, Katja Hofmann, Pierre-Yves Oudeyer
- Abstract summary: We introduce the concept of Meta-ACL, and formalize it in the context of black-box RL learners.
We present AGAIN, a first instantiation of Meta-ACL, and showcase its benefits for curriculum generation over classical ACL.
- Score: 35.13646854355393
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: A major challenge in the Deep RL (DRL) community is to train agents able to
generalize their control policy over situations never seen in training.
Training on diverse tasks has been identified as a key ingredient for good
generalization, which pushed researchers towards using rich procedural task
generation systems controlled through complex continuous parameter spaces. In
such complex task spaces, it is essential to rely on some form of Automatic
Curriculum Learning (ACL) to adapt the task sampling distribution to a given
learning agent, instead of randomly sampling tasks, as many could end up being
either trivial or unfeasible. Since it is hard to get prior knowledge on such
task spaces, many ACL algorithms explore the task space to detect progress
niches over time, a costly tabula-rasa process that needs to be performed for
each new learning agents, although they might have similarities in their
capabilities profiles. To address this limitation, we introduce the concept of
Meta-ACL, and formalize it in the context of black-box RL learners, i.e.
algorithms seeking to generalize curriculum generation to an (unknown)
distribution of learners. In this work, we present AGAIN, a first instantiation
of Meta-ACL, and showcase its benefits for curriculum generation over classical
ACL in multiple simulated environments including procedurally generated parkour
environments with learners of varying morphologies. Videos and code are
available at https://sites.google.com/view/meta-acl .
Related papers
- A Survey of Meta-Reinforcement Learning [69.76165430793571]
We cast the development of better RL algorithms as a machine learning problem itself in a process called meta-RL.
We discuss how, at a high level, meta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each individual task.
We conclude by presenting the open problems on the path to making meta-RL part of the standard toolbox for a deep RL practitioner.
arXiv Detail & Related papers (2023-01-19T12:01:41Z) - Continual Object Detection via Prototypical Task Correlation Guided
Gating Mechanism [120.1998866178014]
We present a flexible framework for continual object detection via pRotOtypical taSk corrElaTion guided gaTingAnism (ROSETTA)
Concretely, a unified framework is shared by all tasks while task-aware gates are introduced to automatically select sub-models for specific tasks.
Experiments on COCO-VOC, KITTI-Kitchen, class-incremental detection on VOC and sequential learning of four tasks show that ROSETTA yields state-of-the-art performance.
arXiv Detail & Related papers (2022-05-06T07:31:28Z) - Fully Online Meta-Learning Without Task Boundaries [80.09124768759564]
We study how meta-learning can be applied to tackle online problems of this nature.
We propose a Fully Online Meta-Learning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries.
Our experiments show that FOML was able to learn new tasks faster than the state-of-the-art online learning methods.
arXiv Detail & Related papers (2022-02-01T07:51:24Z) - Variational Automatic Curriculum Learning for Sparse-Reward Cooperative
Multi-Agent Problems [42.973910399533054]
We introduce a curriculum learning algorithm, Variational Automatic Curriculum Learning (VACL), for solving cooperative multi-agent reinforcement learning problems.
Our VACL algorithm implements this variational paradigm with two practical components, task expansion and entity progression.
Experiment results show that VACL solves a collection of sparse-reward problems with a large number of agents.
arXiv Detail & Related papers (2021-11-08T16:35:08Z) - Learning Multi-Objective Curricula for Deep Reinforcement Learning [55.27879754113767]
Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL)
In this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula.
In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum.
arXiv Detail & Related papers (2021-10-06T19:30:25Z) - TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL [23.719833581321033]
Training autonomous agents able to generalize to multiple tasks is a key target of Deep Reinforcement Learning (DRL) research.
In parallel to improving DRL algorithms, Automatic Curriculum Learning (ACL) study how teacher algorithms can train DRL agents more efficiently by adapting task selection to their evolving abilities.
While multiple standard benchmarks exist to compare DRL agents, there is currently no such thing for ACL algorithms.
arXiv Detail & Related papers (2021-03-17T17:59:22Z) - Curriculum Learning: A Survey [65.31516318260759]
Curriculum learning strategies have been successfully employed in all areas of machine learning.
We construct a taxonomy of curriculum learning approaches by hand, considering various classification criteria.
We build a hierarchical tree of curriculum learning methods using an agglomerative clustering algorithm.
arXiv Detail & Related papers (2021-01-25T20:08:32Z) - Trying AGAIN instead of Trying Longer: Prior Learning for Automatic
Curriculum Learning [39.489869446313065]
A major challenge in the Deep RL (DRL) community is to train agents able to generalize over unseen situations.
We propose a two stage ACL approach where 1) a teacher algorithm first learns to train a DRL agent with a high-exploration curriculum, and then 2) distills learned priors from the first run to generate an "expert curriculum"
Besides demonstrating 50% improvements on average over the current state of the art, the objective of this work is to give a first example of a new research direction oriented towards refining ACL techniques over multiple learners.
arXiv Detail & Related papers (2020-04-07T07:30:27Z) - Curriculum Learning for Reinforcement Learning Domains: A Framework and
Survey [53.73359052511171]
Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback.
We present a framework for curriculum learning (CL) in RL, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals.
arXiv Detail & Related papers (2020-03-10T20:41:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.