SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention

• URL: http://arxiv.org/abs/2312.01990v1
• Date: Mon, 4 Dec 2023 16:08:47 GMT
• Title: SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention
• Authors: Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, Tamas Sarlos, Ken Oslund, Karol Hausman, Kanishka Rao
• Abstract summary: Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT) New paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment.
• Score: 34.276999297736346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment. SARA-RT relies on the new method of fine-tuning proposed by us, called up-training. It converts pre-trained or already fine-tuned Transformer-based robotic policies of quadratic time complexity (including massive billion-parameter vision-language-action models or VLAs), into their efficient linear-attention counterparts maintaining high quality. We demonstrate the effectiveness of SARA-RT by speeding up: (a) the class of recently introduced RT-2 models, the first VLA robotic policies pre-trained on internet-scale data, as well as (b) Point Cloud Transformer (PCT) robotic policies operating on large point clouds. We complement our results with the rigorous mathematical analysis providing deeper insight into the phenomenon of SARA.

Related papers

• VertiFormer: A Data-Efficient Multi-Task Transformer for Off-Road Robot Mobility [49.512339092493384]
  VertiFormer is a novel data-efficient multi-task Transformer model trained with only one hour of data. Our experiments offer insights into effectively utilizing Transformers for off-road robot mobility with limited data.
  arXiv  Detail & Related papers  (2025-02-01T20:21:00Z)
• Force-Based Robotic Imitation Learning: A Two-Phase Approach for Construction Assembly Tasks [2.6092377907704254]
  This paper proposes a two-phase system to improve robot learning. The first phase captures real-time data from operators using a robot arm linked with a virtual simulator via ROS-Sharp. In the second phase, this feedback is converted into robotic motion instructions, using a generative approach to incorporate force feedback into the learning process.
  arXiv  Detail & Related papers  (2025-01-24T22:01:23Z)
• FAST: Efficient Action Tokenization for Vision-Language-Action Models [98.15494168962563]
  We propose a new compression-based tokenization scheme for robot actions, based on the discrete cosine transform. Based on FAST, we release FAST+, a universal robot action tokenizer, trained on 1M real robot action trajectories.
  arXiv  Detail & Related papers  (2025-01-16T18:57:04Z)
• Advances in Transformers for Robotic Applications: A Review [0.9208007322096533]
  We go through recent advances and trends in Transformers in Robotics. We examine their integration into robotic perception, planning, and control for autonomous systems. We discuss how different Transformer variants are being adapted in robotics for reliable planning and perception.
  arXiv  Detail & Related papers  (2024-12-13T23:02:15Z)
• HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers [12.373320641721344]
  Large Vision-Language-Action (VLA) models have shown promise in robotic control due to their impressive generalization ability. Their reliance on VLM backends with billions of parameters leads to high computational costs and latency inference. This paper proposes HiRT, a Hierarchical Robot Transformer framework that enables flexible frequency and performance trade-off.
  arXiv  Detail & Related papers  (2024-09-12T09:18:09Z)
• AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents [109.3804962220498]
  AutoRT is a system to scale up the deployment of operational robots in completely unseen scenarios with minimal human supervision. We demonstrate AutoRT proposing instructions to over 20 robots across multiple buildings and collecting 77k real robot episodes via both teleoperation and autonomous robot policies. We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.
  arXiv  Detail & Related papers  (2024-01-23T18:45:54Z)
• Learning Bipedal Walking for Humanoids with Current Feedback [5.429166905724048]
  We present an approach for overcoming the sim2real gap issue for humanoid robots arising from inaccurate torque-tracking at the actuator level. Our approach successfully trains a unified, end-to-end policy in simulation that can be deployed on a real HRP-5P humanoid robot to achieve bipedal locomotion.
  arXiv  Detail & Related papers  (2023-03-07T08:16:46Z)
• REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfer [57.045140028275036]
  We consider the problem of transferring a policy across two different robots with significantly different parameters such as kinematics and morphology. Existing approaches that train a new policy by matching the action or state transition distribution, including imitation learning methods, fail due to optimal action and/or state distribution being mismatched in different robots. We propose a novel method named $REvolveR$ of using continuous evolutionary models for robotic policy transfer implemented in a physics simulator.
  arXiv  Detail & Related papers  (2022-02-10T18:50:25Z)
• Efficient pre-training objectives for Transformers [84.64393460397471]
  We study several efficient pre-training objectives for Transformers-based models. We prove that eliminating the MASK token and considering the whole output during the loss are essential choices to improve performance.
  arXiv  Detail & Related papers  (2021-04-20T00:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.