Deep reinforcement learning from human preferences

• URL: http://arxiv.org/abs/1706.03741v4
• Date: Fri, 17 Feb 2023 17:00:34 GMT
• Title: Deep reinforcement learning from human preferences
• Authors: Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei
• Abstract summary: We explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems.
• Score: 19.871618959160692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

Related papers

• Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback [27.223725464754853]
  GEAR enables robots to be placed in real-world environments and left to train autonomously without interruption. System streams robot experience to a web interface only requiring occasional asynchronous feedback from remote, crowdsourced, non-expert humans.
  arXiv  Detail & Related papers  (2023-10-31T16:43:56Z)
• REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation [61.7171775202833]
  We introduce an efficient system for learning dexterous manipulation skills withReinforcement learning. The main idea of our approach is the integration of recent advances in sample-efficient RL and replay buffer bootstrapping. Our system completes the real-world training cycle by incorporating learned resets via an imitation-based pickup policy.
  arXiv  Detail & Related papers  (2023-09-06T19:05:31Z)
• Accelerating Interactive Human-like Manipulation Learning with GPU-based Simulation and High-quality Demonstrations [25.393382192511716]
  We present an immersive virtual reality teleoperation interface designed for interactive human-like manipulation on contact rich tasks. We demonstrate the complementary strengths of massively parallel RL and imitation learning, yielding robust and natural behaviors.
  arXiv  Detail & Related papers  (2022-12-05T09:37:27Z)
• A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning [86.06110576808824]
  Deep reinforcement learning is a promising approach to learning policies in uncontrolled environments. Recent advancements in machine learning algorithms and libraries combined with a carefully tuned robot controller lead to learning quadruped in only 20 minutes in the real world.
  arXiv  Detail & Related papers  (2022-08-16T17:37:36Z)
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives [92.0321404272942]
  Reinforcement learning can be used to build general-purpose robotic systems. However, training RL agents to solve robotics tasks still remains challenging. In this work, we manually specify a library of robot action primitives (RAPS), parameterized with arguments that are learned by an RL policy. We find that our simple change to the action interface substantially improves both the learning efficiency and task performance.
  arXiv  Detail & Related papers  (2021-10-28T17:59:30Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned [111.06812202454364]
  We present a number of case studies involving robotic deep RL. We discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting.
  arXiv  Detail & Related papers  (2021-02-04T22:09:28Z)
• Weak Human Preference Supervision For Deep Reinforcement Learning [48.03929962249475]
  The current reward learning from human preferences could be used to resolve complex reinforcement learning (RL) tasks without access to a reward function. We propose a weak human preference supervision framework, for which we developed a human preference scaling model. Our established human-demonstration estimator requires human feedback only for less than 0.01% of the agent's interactions with the environment.
  arXiv  Detail & Related papers  (2020-07-25T10:37:15Z)
• Deep Reinforcement Learning with Interactive Feedback in a Human-Robot Environment [1.2998475032187096]
  We propose a deep reinforcement learning approach with interactive feedback to learn a domestic task in a human-robot scenario. We compare three different learning methods using a simulated robotic arm for the task of organizing different objects. The obtained results show that a learner agent, using either agent-IDeepRL or human-IDeepRL, completes the given task earlier and has fewer mistakes compared to the autonomous DeepRL approach.
  arXiv  Detail & Related papers  (2020-07-07T11:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.