SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation

• URL: http://arxiv.org/abs/2506.15847v1
• Date: Wed, 18 Jun 2025 19:55:10 GMT
• Title: SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation
• Authors: Arpit Bahety, Arnav Balaji, Ben Abbatematteo, Roberto Martín-Martín,
• Abstract summary: SafeMimic is a framework to learn new mobile manipulation skills safely and autonomously from a single third-person human video.<n>It parses the video into segments, inferring both the semantic changes caused and the motions the human executed to achieve them.<n>It then adapts the behavior to the robot's own morphology by sampling candidate actions around the human ones.
• Score: 9.883164852612682
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For robots to become efficient helpers in the home, they must learn to perform new mobile manipulation tasks simply by watching humans perform them. Learning from a single video demonstration from a human is challenging as the robot needs to first extract from the demo what needs to be done and how, translate the strategy from a third to a first-person perspective, and then adapt it to be successful with its own morphology. Furthermore, to mitigate the dependency on costly human monitoring, this learning process should be performed in a safe and autonomous manner. We present SafeMimic, a framework to learn new mobile manipulation skills safely and autonomously from a single third-person human video. Given an initial human video demonstration of a multi-step mobile manipulation task, SafeMimic first parses the video into segments, inferring both the semantic changes caused and the motions the human executed to achieve them and translating them to an egocentric reference. Then, it adapts the behavior to the robot's own morphology by sampling candidate actions around the human ones, and verifying them for safety before execution in a receding horizon fashion using an ensemble of safety Q-functions trained in simulation. When safe forward progression is not possible, SafeMimic backtracks to previous states and attempts a different sequence of actions, adapting both the trajectory and the grasping modes when required for its morphology. As a result, SafeMimic yields a strategy that succeeds in the demonstrated behavior and learns task-specific actions that reduce exploration in future attempts. Our experiments show that our method allows robots to safely and efficiently learn multi-step mobile manipulation behaviors from a single human demonstration, from different users, and in different environments, with improvements over state-of-the-art baselines across seven tasks

Related papers

• DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policy [33.18108154271181]
  We propose DemoDiffusion, a simple and scalable method for enabling robots to perform manipulation tasks in natural environments.<n>Our approach is based on two key insights. First, the hand motion in a human demonstration provides a useful prior for the robot's end-effector trajectory.<n>Second, while this retargeted motion captures the overall structure of the task, it may not align well with plausible robot actions in-context.
  arXiv  Detail & Related papers  (2025-06-25T17:59:01Z)
• ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos [15.809468471562537]
  ZeroMimic generates image goal-conditioned skill policies for several common manipulation tasks.<n>We evaluate ZeroMimic's out-of-the-box performance in varied real-world and simulated kitchen settings.<n>To enable plug-and-play reuse of ZeroMimic policies on other task setups and robots, we release software and policy checkpoints.
  arXiv  Detail & Related papers  (2025-03-31T09:27:00Z)
• Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training [69.54948297520612]
  Learning a generalist embodied agent poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. We introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. Our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-02-22T09:48:47Z)
• Learning Video-Conditioned Policies for Unseen Manipulation Tasks [83.2240629060453]
  Video-conditioned Policy learning maps human demonstrations of previously unseen tasks to robot manipulation skills. We learn our policy to generate appropriate actions given current scene observations and a video of the target task. We validate our approach on a set of challenging multi-task robot manipulation environments and outperform state of the art.
  arXiv  Detail & Related papers  (2023-05-10T16:25:42Z)
• Learning a Universal Human Prior for Dexterous Manipulation from Human Preference [35.54663426598218]
  We propose a framework that learns a universal human prior using direct human preference feedback over videos. A task-agnostic reward model is trained through iteratively generating diverse polices and collecting human preference over the trajectories. Our method empirically demonstrates more human-like behaviors on robot hands in diverse tasks including even unseen tasks.
  arXiv  Detail & Related papers  (2023-04-10T14:17:33Z)
• Zero-Shot Robot Manipulation from Passive Human Videos [59.193076151832145]
  We develop a framework for extracting agent-agnostic action representations from human videos. Our framework is based on predicting plausible human hand trajectories. We deploy the trained model zero-shot for physical robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-02-03T21:39:52Z)
• Cross-Domain Transfer via Semantic Skill Imitation [49.83150463391275]
  We propose an approach for semantic imitation, which uses demonstrations from a source domain, e.g. human videos, to accelerate reinforcement learning (RL) Instead of imitating low-level actions like joint velocities, our approach imitates the sequence of demonstrated semantic skills like "opening the microwave" or "turning on the stove"
  arXiv  Detail & Related papers  (2022-12-14T18:46:14Z)
• HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration [57.045140028275036]
  We show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning. We propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.
  arXiv  Detail & Related papers  (2022-12-08T15:56:13Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Transformers for One-Shot Visual Imitation [28.69615089950047]
  Humans are able to seamlessly visually imitate others, by inferring their intentions and using past experience to achieve the same end goal. Prior research in robot imitation learning has created agents which can acquire diverse skills from expert human operators. This paper investigates techniques which allow robots to partially bridge these domain gaps, using their past experience.
  arXiv  Detail & Related papers  (2020-11-11T18:41:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.