Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach

• URL: http://arxiv.org/abs/2503.12993v1
• Date: Mon, 17 Mar 2025 09:47:42 GMT
• Title: Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach
• Abstract summary: We present Policy Transfer with Online Demonstrations, an active Learning from Demonstration algorithm for policy transfer.<n>We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics.<n>Results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency.
• Score: 3.637365301757111
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator.

Related papers

• A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer [15.906272895307664]
  We explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation.<n>We aim to train a single diffusion policy for block-pushing tasks with multiple shapes.<n>We evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions.
  arXiv  Detail & Related papers  (2026-07-12T19:48:46Z)
• How to Mitigate the Distribution Shift Problem in Robotics Control: A Robust and Adaptive Approach Based on Offline to Online Imitation Learning [15.32457564038423]
  Distribution shift in imitation learning refers to the problem that the agent cannot plan proper actions for a state that has not been visited during the training.<n>This problem can be largely attributed to the inherently narrow state-action coverage provided by expert demonstrations over the full environment.<n>We propose a robust offline to adaptive online imitation learning framework that handles the distribution shift problem in a lifelong, multi-phase scheme.
  arXiv  Detail & Related papers  (2026-05-25T04:30:51Z)
• Action-Free Offline-to-Online RL via Discretised State Policies [8.696862667760778]
  We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of $(s,r,s'$s)$s.<n>We propose learning state policies that recommend desirable next-state transitions rather than actions.
  arXiv  Detail & Related papers  (2026-01-31T09:53:04Z)
• Curating Demonstrations using Online Experience [52.59275477573012]
  We show that Demo-SCORE can effectively identify suboptimal demonstrations without manual curation.<n>Demo-SCORE achieves over 15-35% higher absolute success rate in the resulting policy compared to the base policy trained with all original demonstrations.
  arXiv  Detail & Related papers  (2025-03-05T17:58:16Z)
• Guided Reinforcement Learning for Robust Multi-Contact Loco-Manipulation [12.377289165111028]
  Reinforcement learning (RL) often necessitates a meticulous Markov Decision Process (MDP) design tailored to each task. This work proposes a systematic approach to behavior synthesis and control for multi-contact loco-manipulation tasks. We define a task-independent MDP to train RL policies using only a single demonstration per task generated from a model-based trajectory.
  arXiv  Detail & Related papers  (2024-10-17T17:46:27Z)
• In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks [93.46282380831339]
  In-context learning helps large language models adapt to various tasks by providing demonstrations of the target task. We propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average.
  arXiv  Detail & Related papers  (2024-10-02T13:37:54Z)
• Cross Domain Policy Transfer with Effect Cycle-Consistency [3.3213136251955815]
  Training a robotic policy from scratch using deep reinforcement learning methods can be prohibitively expensive due to sample inefficiency. We propose a novel approach for learning the mapping functions between state and action spaces across domains using unpaired data. Our approach has been tested on three locomotion tasks and two robotic manipulation tasks.
  arXiv  Detail & Related papers  (2024-03-04T13:20:07Z)
• Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning [4.195122359359966]
  Large Language Models (LLMs) are trained on mass knowledge and proved to be equipped with astonishing inference abilities. In this work, we leverage LLMs to understand and profile the system dynamics by a prompt-based grounded action transformation.
  arXiv  Detail & Related papers  (2023-08-28T03:49:13Z)
• IOB: Integrating Optimization Transfer and Behavior Transfer for Multi-Policy Reuse [50.90781542323258]
  Reinforcement learning (RL) agents can transfer knowledge from source policies to a related target task. Previous methods introduce additional components, such as hierarchical policies or estimations of source policies' value functions. We propose a novel transfer RL method that selects the source policy without training extra components.
  arXiv  Detail & Related papers  (2023-08-14T09:22:35Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL [70.01851346635637]
  offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios. We propose to leverage latent-variable policies that can represent a broader class of policy distributions. Our method improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-03-16T21:17:03Z)
• Robust Learning from Observation with Model Misspecification [33.92371002674386]
  Imitation learning (IL) is a popular paradigm for training policies in robotic systems. We propose a robust IL algorithm to learn policies that can effectively transfer to the real environment without fine-tuning.
  arXiv  Detail & Related papers  (2022-02-12T07:04:06Z)
• Human-in-the-Loop Imitation Learning using Remote Teleoperation [72.2847988686463]
  We build a data collection system tailored to 6-DoF manipulation settings. We develop an algorithm to train the policy iteratively on new data collected by the system. We demonstrate that agents trained on data collected by our intervention-based system and algorithm outperform agents trained on an equivalent number of samples collected by non-interventional demonstrators.
  arXiv  Detail & Related papers  (2020-12-12T05:30:35Z)
• Predicting Sim-to-Real Transfer with Probabilistic Dynamics Models [3.7692466417039814]
  We propose a method to predict the sim-to-real transfer performance of RL policies. A probabilistic dynamics model is trained alongside the policy and evaluated on a fixed set of real-world trajectories.
  arXiv  Detail & Related papers  (2020-09-27T15:06:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.