Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward

• URL: http://arxiv.org/abs/2212.01509v1
• Date: Sat, 3 Dec 2022 02:24:59 GMT
• Title: Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
• Authors: Yanjiang Guo, Jingyue Gao, Zheng Wu, Chengming Shi, Jianyu Chen
• Abstract summary: Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new data for every new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Existing LfD methods can not effectively guide learning in mismatched new tasks with sparse rewards.
• Score: 7.51772160511614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Learning from demonstration (LfD) is an effective way to eliminate this problem, which leverages collected expert data to aid online learning. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new data for every new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Such setting can be challenging and we found existing LfD methods can not effectively guide learning in mismatched new tasks with sparse rewards. We propose conservative reward shaping from demonstration (CRSfD), which shapes the sparse rewards using estimated expert value function. To accelerate learning processes, CRSfD guides the agent to conservatively explore around demonstrations. Experimental results of robot manipulation tasks show that our approach outperforms baseline LfD methods when transferring demonstrations collected in a single task to other different but similar tasks.

Related papers

• Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations [24.041217922654738]
  Continuous control problems can be formulated as sparse-reward reinforcement learning (RL) tasks. Online RL methods can automatically explore the state space to solve each new task. However, discovering sequences of actions that lead to a non-zero reward becomes exponentially more difficult as the task horizon increases. We introduce a systematic reward-shaping framework that distills the information contained in 1) a task-agnostic prior data set and 2) a small number of task-specific expert demonstrations.
  arXiv  Detail & Related papers  (2024-12-02T04:37:12Z)
• Efficient Active Imitation Learning with Random Network Distillation [8.517915878774756]
  Random Network Distillation DAgger (RND-DAgger) is a new active imitation learning method. It limits expert querying by using a learned state-based out-of-distribution measure to trigger interventions. We evaluate RND-DAgger against traditional imitation learning and other active approaches in 3D video games and in a robotic task.
  arXiv  Detail & Related papers  (2024-11-04T08:50:52Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe is a framework that combines the strengths of imitation learning and inverse reinforcement learning to learn a dense reward function efficiently. Our framework produces high-performing policies in high-dimensional tasks where direct imitation fails to replicate complex behaviors.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks [8.320969283401233]
  We show that the standard, naive approach to exploration can manifest as a suboptimal local maximum. We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of multiple exploratory, auxiliary tasks.
  arXiv  Detail & Related papers  (2022-12-30T20:38:54Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Divide & Conquer Imitation Learning [75.31752559017978]
  Imitation Learning can be a powerful approach to bootstrap the learning process. We present a novel algorithm designed to imitate complex robotic tasks from the states of an expert trajectory. We show that our method imitates a non-holonomic navigation task and scales to a complex simulated robotic manipulation task with very high sample efficiency.
  arXiv  Detail & Related papers  (2022-04-15T09:56:50Z)
• Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning [7.51557557629519]
  We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of, in addition to a main task, multiple auxiliary tasks. This affords many benefits: learning efficiency is improved for main tasks with challenging bottleneck transitions, expert data becomes reusable between tasks, and transfer learning through the reuse of learned auxiliary task models becomes possible.
  arXiv  Detail & Related papers  (2021-12-16T14:58:08Z)
• Automatic Curricula via Expert Demonstrations [6.651864489482536]
  We propose Automatic Curricula via Expert Demonstrations (ACED) as a reinforcement learning (RL) approach. ACED extracts curricula from expert demonstration trajectories by dividing demonstrations into sections and initializing training episodes to states sampled from different sections of demonstrations. We show that a combination of ACED with behavior cloning allows pick-and-place tasks to be learned with as few as 1 demonstration and block stacking tasks to be learned with 20 demonstrations.
  arXiv  Detail & Related papers  (2021-06-16T22:21:09Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.