Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics

• URL: http://arxiv.org/abs/2403.19578v3
• Date: Thu, 17 Oct 2024 19:31:39 GMT
• Title: Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics
• Authors: Norman Di Palo, Edward Johns,
• Abstract summary: We show that off-the-shelf text-based Transformers, with no additional training, can perform few-shot in-context visual imitation learning. We achieve this by transforming visual observations into sequences of tokens that a text-pretrained Transformer can ingest and generate. Despite being trained only on language, we show that these Transformers excel at translating tokenised visual keypoint observations into action trajectories.
• Score: 11.88216611522207
• License:
• Abstract: We show that off-the-shelf text-based Transformers, with no additional training, can perform few-shot in-context visual imitation learning, mapping visual observations to action sequences that emulate the demonstrator's behaviour. We achieve this by transforming visual observations (inputs) and trajectories of actions (outputs) into sequences of tokens that a text-pretrained Transformer (GPT-4 Turbo) can ingest and generate, via a framework we call Keypoint Action Tokens (KAT). Despite being trained only on language, we show that these Transformers excel at translating tokenised visual keypoint observations into action trajectories, performing on par or better than state-of-the-art imitation learning (diffusion policies) in the low-data regime on a suite of real-world, everyday tasks. Rather than operating in the language domain as is typical, KAT leverages text-based Transformers to operate in the vision and action domains to learn general patterns in demonstration data for highly efficient imitation learning, indicating promising new avenues for repurposing natural language models for embodied tasks. Videos are available at https://www.robot-learning.uk/keypoint-action-tokens.

Related papers

• Latent Action Pretraining from Videos [156.88613023078778]
  We introduce Latent Action Pretraining for general Action models (LAPA) LAPA is an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels. We propose a method to learn from internet-scale videos that do not have robot action labels.
  arXiv  Detail & Related papers  (2024-10-15T16:28:09Z)
• FILS: Self-Supervised Video Feature Prediction In Semantic Language Space [11.641926922266347]
  This paper demonstrates a self-supervised approach for learning semantic video representations. We present FILS, a novel self-supervised video Feature prediction In semantic Language Space.
  arXiv  Detail & Related papers  (2024-06-05T16:44:06Z)
• Any-point Trajectory Modeling for Policy Learning [64.23861308947852]
  We introduce Any-point Trajectory Modeling (ATM) to predict future trajectories of arbitrary points within a video frame. ATM outperforms strong video pre-training baselines by 80% on average. We show effective transfer learning of manipulation skills from human videos and videos from a different robot morphology.
  arXiv  Detail & Related papers  (2023-12-28T23:34:43Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• Learning Expressive Prompting With Residuals for Vision Transformers [11.342913284654706]
  We present Expressive Prompts with Residuals (EXPRES) which modifies the prompt learning paradigm specifically for effective adaptation of vision transformers (ViT) We apply EXPRES for image classification, few shot learning, and semantic segmentation, and show our method is capable of achieving state of the art prompt tuning on 3/3 categories of the VTAB benchmark.
  arXiv  Detail & Related papers  (2023-03-27T20:47:01Z)
• Instruction-Following Agents with Multimodal Transformer [95.70039658112873]
  We propose a simple yet effective model for robots to solve instruction-following tasks in vision-based environments. Our method consists of a multimodal transformer that encodes visual observations and language instructions. We show that this unified transformer model outperforms all state-of-the-art pre-trained or trained-from-scratch methods in both single-task and multi-task settings.
  arXiv  Detail & Related papers  (2022-10-24T17:46:47Z)
• Episodic Transformer for Vision-and-Language Navigation [142.6236659368177]
  This paper focuses on addressing two challenges: handling long sequence of subtasks, and understanding complex human instructions. We propose Episodic Transformer (E.T.), a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions. Our approach sets a new state of the art on the challenging ALFRED benchmark, achieving 38.4% and 8.5% task success rates on seen and unseen test splits.
  arXiv  Detail & Related papers  (2021-05-13T17:51:46Z)
• Self-Supervised learning with cross-modal transformers for emotion recognition [20.973999078271483]
  Self-supervised learning has shown improvements on tasks with limited labeled datasets in domains like speech and natural language. In this work, we extend self-supervised training to multi-modal applications.
  arXiv  Detail & Related papers  (2020-11-20T21:38:34Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.