DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks

• URL: http://arxiv.org/abs/2404.16779v1
• Date: Thu, 25 Apr 2024 17:28:33 GMT
• Title: DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks
• Authors: Tongzhou Mu, Minghua Liu, Hao Su,
• Abstract summary: We propose DrS (Dense reward learning from Stages), a novel approach for learning reusable dense rewards for multi-stage tasks. By leveraging the stage structures of the task, DrS learns a high-quality dense reward from sparse rewards and demonstrations if given. Experiments on three physical robot manipulation task families with 1000+ task variants demonstrate that our learned rewards can be reused in unseen tasks.
• Score: 26.730889757506915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The success of many RL techniques heavily relies on human-engineered dense rewards, which typically demand substantial domain expertise and extensive trial and error. In our work, we propose DrS (Dense reward learning from Stages), a novel approach for learning reusable dense rewards for multi-stage tasks in a data-driven manner. By leveraging the stage structures of the task, DrS learns a high-quality dense reward from sparse rewards and demonstrations if given. The learned rewards can be \textit{reused} in unseen tasks, thus reducing the human effort for reward engineering. Extensive experiments on three physical robot manipulation task families with 1000+ task variants demonstrate that our learned rewards can be reused in unseen tasks, resulting in improved performance and sample efficiency of RL algorithms. The learned rewards even achieve comparable performance to human-engineered rewards on some tasks. See our project page (https://sites.google.com/view/iclr24drs) for more details.

Related papers

• On-Robot Reinforcement Learning with Goal-Contrastive Rewards [24.415607337006968]
  Reinforcement Learning (RL) has the potential to enable robots to learn from their own actions in the real world. We propose GCR (Goal-intensiveive Rewards), a dense reward function learning method that can be trained on passive video demonstrations. GCR combines two loss functions, an implicit value loss function that models how the reward increases when traversing a successful trajectory, and a goal-contrastive loss that discriminates between successful and failed trajectories.
  arXiv  Detail & Related papers  (2024-10-25T22:11:54Z)
• Affordance-Guided Reinforcement Learning via Visual Prompting [51.361977466993345]
  Keypoint-based Affordance Guidance for Improvements (KAGI) is a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL. On real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 20K online fine-tuning steps.
  arXiv  Detail & Related papers  (2024-07-14T21:41:29Z)
• Go Beyond Imagination: Maximizing Episodic Reachability with World Models [68.91647544080097]
  In this paper, we introduce a new intrinsic reward design called GoBI - Go Beyond Imagination. We apply learned world models to generate predicted future states with random actions. Our method greatly outperforms previous state-of-the-art methods on 12 of the most challenging Minigrid navigation tasks.
  arXiv  Detail & Related papers  (2023-08-25T20:30:20Z)
• Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems [111.80916118530398]
  reinforcement learning (RL) techniques can naturally be utilized to train dialogue strategies to achieve user-specific goals. This paper aims at answering the question of how to efficiently learn and leverage a reward function for training end-to-end (E2E) ToD agents.
  arXiv  Detail & Related papers  (2023-02-20T22:10:04Z)
• Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward [7.51772160511614]
  Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new data for every new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Existing LfD methods can not effectively guide learning in mismatched new tasks with sparse rewards.
  arXiv  Detail & Related papers  (2022-12-03T02:24:59Z)
• Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control [9.118706387430883]
  Reinforcement learning (RL) has recently proven great success in various domains. Yet, the design of the reward function requires detailed domain expertise and tedious fine-tuning to ensure that agents are able to learn the desired behaviour. We propose to use model predictive control(MPC) as an experience source for training RL agents in sparse reward environments.
  arXiv  Detail & Related papers  (2022-10-04T11:06:38Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Lipschitz-constrained Unsupervised Skill Discovery [91.51219447057817]
  Lipschitz-constrained Skill Discovery (LSD) encourages the agent to discover more diverse, dynamic, and far-reaching skills. LSD outperforms previous approaches in terms of skill diversity, state space coverage, and performance on seven downstream tasks.
  arXiv  Detail & Related papers  (2022-02-02T08:29:04Z)
• Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning [7.51557557629519]
  We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of, in addition to a main task, multiple auxiliary tasks. This affords many benefits: learning efficiency is improved for main tasks with challenging bottleneck transitions, expert data becomes reusable between tasks, and transfer learning through the reuse of learned auxiliary task models becomes possible.
  arXiv  Detail & Related papers  (2021-12-16T14:58:08Z)
• A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learning [19.67628391301068]
  We study the performance of multiple state-of-the-art deep reinforcement learning algorithms under different types of reward. Our results show that visual dense rewards are more successful than visual sparse rewards and that there is no single best algorithm for all tasks.
  arXiv  Detail & Related papers  (2021-08-06T17:47:48Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.