It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation

• URL: http://arxiv.org/abs/2202.10608v1
• Date: Tue, 22 Feb 2022 01:23:23 GMT
• Title: It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation
• Authors: Yuqing Du, Pieter Abbeel, Aditya Grover
• Abstract summary: Training general-purpose reinforcement learning agents efficiently requires automatic generation of a goal curriculum. We propose Curriculum Self Play (CuSP), an automated goal generation framework. We demonstrate that our method succeeds at generating an effective curricula of goals for a range of control tasks.
• Score: 107.10235120286352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals. Training such agents efficiently requires automatic generation of a goal curriculum. This is challenging as it requires (a) exploring goals of increasing difficulty, while ensuring that the agent (b) is exposed to a diverse set of goals in a sample efficient manner and (c) does not catastrophically forget previously solved goals. We propose Curriculum Self Play (CuSP), an automated goal generation framework that seeks to satisfy these desiderata by virtue of a multi-player game with four agents. We extend the asymmetric curricula learning in PAIRED (Dennis et al., 2020) to a symmetrized game that carefully balances cooperation and competition between two off-policy student learners and two regret-maximizing teachers. CuSP additionally introduces entropic goal coverage and accounts for the non-stationary nature of the students, allowing us to automatically induce a curriculum that balances progressive exploration with anti-catastrophic exploitation. We demonstrate that our method succeeds at generating an effective curricula of goals for a range of control tasks, outperforming other methods at zero-shot test-time generalization to novel out-of-distribution goals.

Related papers

• Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning [42.540853953923495]
  We introduce a novel automatic curriculum learning framework, Skilled Population Curriculum (SPC), which adapts curriculum learning to multi-agent coordination. Specifically, we endow the student with population-invariant communication and a hierarchical skill set, allowing it to learn cooperation and behavior skills from distinct tasks with varying numbers of agents. We also analyze the inherent non-stationarity of this multi-agent automatic curriculum teaching problem and provide a corresponding regret bound.
  arXiv  Detail & Related papers  (2023-02-07T12:30:52Z)
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration [68.94506047556412]
  We propose to leverage a sequential bias to learn control policies for complex robotic tasks using a single demonstration. We show that DCIL-II can solve with unprecedented sample efficiency some challenging simulated tasks such as humanoid locomotion and stand-up.
  arXiv  Detail & Related papers  (2022-11-09T10:28:40Z)
• Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning [99.38163119531745]
  We show that applying a discretizing bottleneck can improve performance in goal-conditioned RL setups. We experimentally prove the expected return on out-of-distribution goals, while still allowing for specifying goals with expressive structure.
  arXiv  Detail & Related papers  (2022-11-01T03:31:43Z)
• Automatic Goal Generation using Dynamical Distance Learning [5.797847756967884]
  Reinforcement Learning (RL) agents can learn to solve complex sequential decision making tasks by interacting with the environment. In the field of multi-goal RL, where agents are required to reach multiple goals to solve complex tasks, improving sample efficiency can be especially challenging. We propose a method for automatic goal generation using a dynamical distance function (DDF) in a self-supervised fashion.
  arXiv  Detail & Related papers  (2021-11-07T16:23:56Z)
• Adversarial Intrinsic Motivation for Reinforcement Learning [60.322878138199364]
  We investigate whether the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution can be utilized effectively for reinforcement learning tasks. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function.
  arXiv  Detail & Related papers  (2021-05-27T17:51:34Z)
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals [63.680207855344875]
  AMIGo is a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks.
  arXiv  Detail & Related papers  (2020-06-22T10:22:08Z)
• Automatic Curriculum Learning through Value Disagreement [95.19299356298876]
  Continually solving new, unsolved tasks is the key to learning diverse behaviors. In the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. We propose setting up an automatic curriculum for goals that the agent needs to solve. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T03:58:25Z)
• Generating Automatic Curricula via Self-Supervised Active Domain Randomization [11.389072560141388]
  We extend the self-play framework to jointly learn a goal and environment curriculum. Our method generates a coupled goal-task curriculum, where agents learn through progressively more difficult tasks and environment variations. Our results show that a curriculum of co-evolving the environment difficulty together with the difficulty of goals set in each environment provides practical benefits in the goal-directed tasks tested.
  arXiv  Detail & Related papers  (2020-02-18T22:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.