Physically Embedded Planning Problems: New Challenges for Reinforcement Learning

• URL: http://arxiv.org/abs/2009.05524v2
• Date: Thu, 29 Oct 2020 17:28:38 GMT
• Title: Physically Embedded Planning Problems: New Challenges for Reinforcement Learning
• Authors: Mehdi Mirza, Andrew Jaegle, Jonathan J. Hunt, Arthur Guez, Saran Tunyasuvunakool, Alistair Muldal, Th\'eophane Weber, Peter Karkus, S\'ebastien Racani\`ere, Lars Buesing, Timothy Lillicrap, Nicolas Heess
• Abstract summary: Recent work in deep reinforcement learning (RL) has produced algorithms capable of mastering challenging games such as Go, chess, or shogi. We introduce a set of physically embedded planning problems and make them publicly available. We find that existing RL algorithms struggle to master even the simplest of their physically embedded counterparts.
• Score: 26.74526714574981
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work in deep reinforcement learning (RL) has produced algorithms capable of mastering challenging games such as Go, chess, or shogi. In these works the RL agent directly observes the natural state of the game and controls that state directly with its actions. However, when humans play such games, they do not just reason about the moves but also interact with their physical environment. They understand the state of the game by looking at the physical board in front of them and modify it by manipulating pieces using touch and fine-grained motor control. Mastering complicated physical systems with abstract goals is a central challenge for artificial intelligence, but it remains out of reach for existing RL algorithms. To encourage progress towards this goal we introduce a set of physically embedded planning problems and make them publicly available. We embed challenging symbolic tasks (Sokoban, tic-tac-toe, and Go) in a physics engine to produce a set of tasks that require perception, reasoning, and motor control over long time horizons. Although existing RL algorithms can tackle the symbolic versions of these tasks, we find that they struggle to master even the simplest of their physically embedded counterparts. As a first step towards characterizing the space of solution to these tasks, we introduce a strong baseline that uses a pre-trained expert game player to provide hints in the abstract space to an RL agent's policy while training it on the full sensorimotor control task. The resulting agent solves many of the tasks, underlining the need for methods that bridge the gap between abstract planning and embodied control. See illustrating video at https://youtu.be/RwHiHlym_1k.

Related papers

• Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks [3.479490713357225]
  We procedurally generate tens of millions of 2D physics-based tasks and use these to train a general reinforcement learning (RL) agent for physical control. Kinetix is an open-ended space of physics-based RL environments that can represent tasks ranging from robotic locomotion and grasping to video games and classic RL environments. Our trained agent exhibits strong physical reasoning capabilities, being able to zero-shot solve unseen human-designed environments.
  arXiv  Detail & Related papers  (2024-10-30T16:59:41Z)
• Mastering the Game of Guandan with Deep Reinforcement Learning and Behavior Regulating [16.718186690675164]
  We propose a framework named GuanZero for AI agents to master the game of Guandan. The main contribution of this paper is about regulating agents' behavior through a carefully designed neural network encoding scheme.
  arXiv  Detail & Related papers  (2024-02-21T07:26:06Z)
• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data [101.43350024175157]
  Self-supervised learning has the potential to decrease the amount of human annotation and engineering effort required to learn control strategies. Our work builds on prior work showing that the reinforcement learning (RL) itself can be cast as a self-supervised problem. We demonstrate that a self-supervised RL algorithm based on contrastive learning can solve real-world, image-based robotic manipulation tasks.
  arXiv  Detail & Related papers  (2023-06-06T01:36:56Z)
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives [92.0321404272942]
  Reinforcement learning can be used to build general-purpose robotic systems. However, training RL agents to solve robotics tasks still remains challenging. In this work, we manually specify a library of robot action primitives (RAPS), parameterized with arguments that are learned by an RL policy. We find that our simple change to the action interface substantially improves both the learning efficiency and task performance.
  arXiv  Detail & Related papers  (2021-10-28T17:59:30Z)
• From Motor Control to Team Play in Simulated Humanoid Football [56.86144022071756]
  We train teams of physically simulated humanoid avatars to play football in a realistic virtual environment. In a sequence of stages, players first learn to control a fully articulated body to perform realistic, human-like movements. They then acquire mid-level football skills such as dribbling and shooting. Finally, they develop awareness of others and play as a team, bridging the gap between low-level motor control at a timescale of milliseconds.
  arXiv  Detail & Related papers  (2021-05-25T20:17:10Z)
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned [111.06812202454364]
  We present a number of case studies involving robotic deep RL. We discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting.
  arXiv  Detail & Related papers  (2021-02-04T22:09:28Z)
• The NetHack Learning Environment [79.06395964379107]
  We present the NetHack Learning Environment (NLE), a procedurally generated rogue-like environment for Reinforcement Learning research. We argue that NetHack is sufficiently complex to drive long-term research on problems such as exploration, planning, skill acquisition, and language-conditioned RL. We demonstrate empirical success for early stages of the game using a distributed Deep RL baseline and Random Network Distillation exploration.
  arXiv  Detail & Related papers  (2020-06-24T14:12:56Z)
• Learning to Play Table Tennis From Scratch using Muscular Robots [34.34824536814943]
  This work is the first to (a) fail-safe learn of a safety-critical dynamic task using anthropomorphic robot arms, (b) learn a precision-demanding problem with a PAM-driven system, and (c) train robots to play table tennis without real balls. Videos and datasets are available at muscularTT.embodied.ml.
  arXiv  Detail & Related papers  (2020-06-10T16:43:27Z)
• Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks [70.56451186797436]
  We study how to use meta-reinforcement learning to solve the bulk of the problem in simulation. We demonstrate our approach by training an agent to successfully perform challenging real-world insertion tasks.
  arXiv  Detail & Related papers  (2020-04-29T18:00:22Z)
• Deep Adversarial Reinforcement Learning for Object Disentangling [36.66974848126079]
  We present a novel adversarial reinforcement learning (ARL) framework for disentangling waste objects. The ARL framework utilizes an adversary, which is trained to steer the original agent, the protagonist, to challenging states. We show that our method can generalize from training to test scenarios by training an end-to-end system for robot control to solve a challenging object disentangling task.
  arXiv  Detail & Related papers  (2020-03-08T13:20:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.