Instant Policy: In-Context Imitation Learning via Graph Diffusion

• URL: http://arxiv.org/abs/2411.12633v1
• Date: Tue, 19 Nov 2024 16:45:52 GMT
• Title: Instant Policy: In-Context Imitation Learning via Graph Diffusion
• Authors: Vitalis Vosylius, Edward Johns,
• Abstract summary: In-context Imitation Learning (ICIL) is a promising opportunity for robotics. We introduce Instant Policy, which learns new tasks instantly from just one or two demonstrations. We also show how it can serve as a foundation for cross-embodiment and zero-shot transfer to language-defined tasks.
• Score: 12.879700241782528
• License:
• Abstract: Following the impressive capabilities of in-context learning with large transformers, In-Context Imitation Learning (ICIL) is a promising opportunity for robotics. We introduce Instant Policy, which learns new tasks instantly (without further training) from just one or two demonstrations, achieving ICIL through two key components. First, we introduce inductive biases through a graph representation and model ICIL as a graph generation problem with a learned diffusion process, enabling structured reasoning over demonstrations, observations, and actions. Second, we show that such a model can be trained using pseudo-demonstrations - arbitrary trajectories generated in simulation - as a virtually infinite pool of training data. Simulated and real experiments show that Instant Policy enables rapid learning of various everyday robot tasks. We also show how it can serve as a foundation for cross-embodiment and zero-shot transfer to language-defined tasks. Code and videos are available at https://www.robot-learning.uk/instant-policy.

Related papers

• Flow as the Cross-Domain Manipulation Interface [73.15952395641136]
  Im2Flow2Act enables robots to acquire real-world manipulation skills without the need of real-world robot training data. Im2Flow2Act comprises two components: a flow generation network and a flow-conditioned policy. We demonstrate Im2Flow2Act's capabilities in a variety of real-world tasks, including the manipulation of rigid, articulated, and deformable objects.
  arXiv  Detail & Related papers  (2024-07-21T16:15:02Z)
• LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning [50.99807031490589]
  We introduce LLARVA, a model trained with a novel instruction tuning method to unify a range of robotic learning tasks, scenarios, and environments. We generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model. Experiments yield strong performance, demonstrating that LLARVA performs well compared to several contemporary baselines.
  arXiv  Detail & Related papers  (2024-06-17T17:55:29Z)
• DITTO: Demonstration Imitation by Trajectory Transformation [31.930923345163087]
  In this work, we address the problem of one-shot imitation from a single human demonstration, given by an RGB-D video recording. We propose a two-stage process. In the first stage we extract the demonstration trajectory offline. This entails segmenting manipulated objects and determining their relative motion in relation to secondary objects such as containers. In the online trajectory generation stage, we first re-detect all objects, then warp the demonstration trajectory to the current scene and execute it on the robot.
  arXiv  Detail & Related papers  (2024-03-22T13:46:51Z)
• Any-point Trajectory Modeling for Policy Learning [64.23861308947852]
  We introduce Any-point Trajectory Modeling (ATM) to predict future trajectories of arbitrary points within a video frame. ATM outperforms strong video pre-training baselines by 80% on average. We show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology.
  arXiv  Detail & Related papers  (2023-12-28T23:34:43Z)
• Learning to Act from Actionless Videos through Dense Correspondences [87.1243107115642]
  We present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments. Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as a general representation for specifying robot goals. We demonstrate the efficacy of our approach in learning policies on table-top manipulation and navigation tasks.
  arXiv  Detail & Related papers  (2023-10-12T17:59:23Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• Continual Learning from Demonstration of Robotics Skills [5.573543601558405]
  Methods for teaching motion skills to robots focus on training for a single skill at a time. We propose an approach for continual learning from demonstration using hypernetworks and neural ordinary differential equation solvers.
  arXiv  Detail & Related papers  (2022-02-14T16:26:52Z)
• Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms [16.939129935919325]
  Video2Skill (V2S) attempts to extend this capability to artificial agents by allowing a robot arm to learn from human cooking videos. We first use sequence-to-sequence Auto-Encoder style architectures to learn a temporal latent space for events in long-horizon demonstrations. We then transfer these representations to the robotic target domain, using a small amount of offline and unrelated interaction data.
  arXiv  Detail & Related papers  (2021-09-08T17:59:01Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.