Robotic Imitation of Human Actions

• URL: http://arxiv.org/abs/2401.08381v2
• Date: Mon, 3 Jun 2024 13:40:44 GMT
• Title: Robotic Imitation of Human Actions
• Authors: Josua Spisak, Matthias Kerzel, Stefan Wermter,
• Abstract summary: We introduce a new approach to imitation learning that tackles the challenges of a robot imitating a human. Our approach can use a single human demonstration to abstract information about the demonstrated task, and use that information to generalise and replicate it.
• Score: 16.26334759935617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imitation can allow us to quickly gain an understanding of a new task. Through a demonstration, we can gain direct knowledge about which actions need to be performed and which goals they have. In this paper, we introduce a new approach to imitation learning that tackles the challenges of a robot imitating a human, such as the change in perspective and body schema. Our approach can use a single human demonstration to abstract information about the demonstrated task, and use that information to generalise and replicate it. We facilitate this ability by a new integration of two state-of-the-art methods: a diffusion action segmentation model to abstract temporal information from the demonstration and an open vocabulary object detector for spatial information. Furthermore, we refine the abstracted information and use symbolic reasoning to create an action plan utilising inverse kinematics, to allow the robot to imitate the demonstrated action.

Related papers

• Learning Goal-oriented Bimanual Dough Rolling Using Dynamic Heterogeneous Graph Based on Human Demonstration [19.74767906744719]
  Soft object manipulation poses significant challenges for robots, requiring effective techniques for state representation and manipulation policy learning. This research paper introduces a novel approach: a dynamic heterogeneous graph-based model for learning goal-oriented soft object manipulation policies.
  arXiv  Detail & Related papers  (2024-10-15T16:12:00Z)
• Self-Explainable Affordance Learning with Embodied Caption [63.88435741872204]
  We introduce Self-Explainable Affordance learning (SEA) with embodied caption. SEA enables robots to articulate their intentions and bridge the gap between explainable vision-language caption and visual affordance learning. We propose a novel model to effectively combine affordance grounding with self-explanation in a simple but efficient manner.
  arXiv  Detail & Related papers  (2024-04-08T15:22:38Z)
• BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning [108.41464483878683]
  We study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks. We develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%.
  arXiv  Detail & Related papers  (2022-02-04T07:30:48Z)
• Procedure Planning in Instructional Videosvia Contextual Modeling and Model-based Policy Learning [114.1830997893756]
  This work focuses on learning a model to plan goal-directed actions in real-life videos. We propose novel algorithms to model human behaviors through Bayesian Inference and model-based Imitation Learning.
  arXiv  Detail & Related papers  (2021-10-05T01:06:53Z)
• Property-Aware Robot Object Manipulation: a Generative Approach [57.70237375696411]
  In this work, we focus on how to generate robot motion adapted to the hidden properties of the manipulated objects. We explore the possibility of leveraging Generative Adversarial Networks to synthesize new actions coherent with the properties of the object. Our results show that Generative Adversarial Nets can be a powerful tool for the generation of novel and meaningful transportation actions.
  arXiv  Detail & Related papers  (2021-06-08T14:15:36Z)
• Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration [8.57914821832517]
  We introduce a simple new method for visual imitation learning, which allows a novel robot manipulation task to be learned from a single human demonstration. Our method models imitation learning as a state estimation problem, with the state defined as the end-effector's pose. At test time, the end-effector moves to the estimated state through a linear path, at which point the original demonstration's end-effector velocities are simply replayed.
  arXiv  Detail & Related papers  (2021-05-13T16:36:55Z)
• Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos [28.712673809577076]
  We present an approach for physical imitation from human videos for robot manipulation tasks. We design a perception module that learns to translate human videos to the robot domain followed by unsupervised keypoint detection. We evaluate the effectiveness of our approach on five robot manipulation tasks, including reaching, pushing, sliding, coffee making, and drawer closing.
  arXiv  Detail & Related papers  (2021-01-18T18:50:32Z)
• Learning Object Manipulation Skills via Approximate State Estimation from Real Videos [47.958512470724926]
  Humans are adept at learning new tasks by watching a few instructional videos. On the other hand, robots that learn new actions either require a lot of effort through trial and error, or use expert demonstrations that are challenging to obtain. In this paper, we explore a method that facilitates learning object manipulation skills directly from videos.
  arXiv  Detail & Related papers  (2020-11-13T08:53:47Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)
• Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream [4.450615100675747]
  We present a vision dataset under a strictly constrained knowledge domain for both robot and human manipulations. We propose a scheme to generate a combination of visual attentions and an evolving knowledge graph filled with commonsense knowledge. The proposed scheme allows the robot to mimic human-like intentional behaviors by watching real-time videos.
  arXiv  Detail & Related papers  (2020-03-02T19:48:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.