DayDreamer: World Models for Physical Robot Learning

• URL: http://arxiv.org/abs/2206.14176v1
• Date: Tue, 28 Jun 2022 17:44:48 GMT
• Title: DayDreamer: World Models for Physical Robot Learning
• Authors: Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, Pieter Abbeel
• Abstract summary: Deep reinforcement learning is a common approach to robot learning but requires a large amount of trial and error to learn. Many advances in robot learning rely on simulators. In this paper, we apply Dreamer to 4 robots to learn online and directly in the real world, without simulators.
• Score: 142.11031132529524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To solve tasks in complex environments, robots need to learn from experience. Deep reinforcement learning is a common approach to robot learning but requires a large amount of trial and error to learn, limiting its deployment in the physical world. As a consequence, many advances in robot learning rely on simulators. On the other hand, learning inside of simulators fails to capture the complexity of the real world, is prone to simulator inaccuracies, and the resulting behaviors do not adapt to changes in the world. The Dreamer algorithm has recently shown great promise for learning from small amounts of interaction by planning within a learned world model, outperforming pure reinforcement learning in video games. Learning a world model to predict the outcomes of potential actions enables planning in imagination, reducing the amount of trial and error needed in the real environment. However, it is unknown whether Dreamer can facilitate faster learning on physical robots. In this paper, we apply Dreamer to 4 robots to learn online and directly in the real world, without simulators. Dreamer trains a quadruped robot to roll off its back, stand up, and walk from scratch and without resets in only 1 hour. We then push the robot and find that Dreamer adapts within 10 minutes to withstand perturbations or quickly roll over and stand back up. On two different robotic arms, Dreamer learns to pick and place multiple objects directly from camera images and sparse rewards, approaching human performance. On a wheeled robot, Dreamer learns to navigate to a goal position purely from camera images, automatically resolving ambiguity about the robot orientation. Using the same hyperparameters across all experiments, we find that Dreamer is capable of online learning in the real world, establishing a strong baseline. We release our infrastructure for future applications of world models to robot learning.

Related papers

• Structured World Models from Human Videos [45.08503470821952]
  We tackle the problem of learning complex, general behaviors directly in the real world. We propose an approach for robots to efficiently learn manipulation skills using only a handful of real-world interaction trajectories.
  arXiv  Detail & Related papers  (2023-08-21T17:59:32Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Affordances from Human Videos as a Versatile Representation for Robotics [31.248842798600606]
  We train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show the efficacy of our approach, which we call VRB, across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild.
  arXiv  Detail & Related papers  (2023-04-17T17:59:34Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots [87.32145104894754]
  We introduce a framework for training generalized locomotion (GenLoco) controllers for quadrupedal robots. Our framework synthesizes general-purpose locomotion controllers that can be deployed on a large variety of quadrupedal robots. We show that our models acquire more general control strategies that can be directly transferred to novel simulated and real-world robots.
  arXiv  Detail & Related papers  (2022-09-12T15:14:32Z)
• Back to Reality for Imitation Learning [8.57914821832517]
  Imitation learning, and robot learning in general, emerged due to breakthroughs in machine learning, rather than breakthroughs in robotics. We believe that a better metric for real-world robot learning is time efficiency, which better models the true cost to humans.
  arXiv  Detail & Related papers  (2021-11-25T02:03:52Z)
• Lifelong Robotic Reinforcement Learning by Retaining Experiences [61.79346922421323]
  Many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In this work, we study a practical sequential multi-task RL problem motivated by the practical constraints of physical robotic systems. We derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set.
  arXiv  Detail & Related papers  (2021-09-19T18:00:51Z)
• Continual Learning of Visual Concepts for Robots through Limited Supervision [9.89901717499058]
  My research focuses on developing robots that continually learn in dynamic unseen environments/scenarios. I develop machine learning models that produce State-of-the-results on benchmark datasets.
  arXiv  Detail & Related papers  (2021-01-26T01:26:07Z)
• A Survey of Behavior Learning Applications in Robotics -- State of the Art and Perspectives [44.45953630612019]
  Recent success of machine learning in many domains has been overwhelming. We will give a broad overview of behaviors that have been learned and used on real robots.
  arXiv  Detail & Related papers  (2019-06-05T07:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.