Sim-to-Real Transfer for Quadrupedal Locomotion via Terrain Transformer

• URL: http://arxiv.org/abs/2212.07740v2
• Date: Tue, 21 Mar 2023 06:06:45 GMT
• Title: Sim-to-Real Transfer for Quadrupedal Locomotion via Terrain Transformer
• Authors: Hang Lai, Weinan Zhang, Xialin He, Chen Yu, Zheng Tian, Yong Yu, Jun Wang
• Abstract summary: We propose a high-capacity Transformer model for quadrupedal locomotion control on various terrains. To better leverage Transformer in sim-to-real scenarios, we present a novel two-stage training framework consisting of an offline pretraining stage and an online correction stage. Experiments in simulation demonstrate that TERT outperforms state-of-the-art baselines on different terrains in terms of return, energy consumption and control smoothness.
• Score: 31.581743045813557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep reinforcement learning has recently emerged as an appealing alternative for legged locomotion over multiple terrains by training a policy in physical simulation and then transferring it to the real world (i.e., sim-to-real transfer). Despite considerable progress, the capacity and scalability of traditional neural networks are still limited, which may hinder their applications in more complex environments. In contrast, the Transformer architecture has shown its superiority in a wide range of large-scale sequence modeling tasks, including natural language processing and decision-making problems. In this paper, we propose Terrain Transformer (TERT), a high-capacity Transformer model for quadrupedal locomotion control on various terrains. Furthermore, to better leverage Transformer in sim-to-real scenarios, we present a novel two-stage training framework consisting of an offline pretraining stage and an online correction stage, which can naturally integrate Transformer with privileged training. Extensive experiments in simulation demonstrate that TERT outperforms state-of-the-art baselines on different terrains in terms of return, energy consumption and control smoothness. In further real-world validation, TERT successfully traverses nine challenging terrains, including sand pit and stair down, which can not be accomplished by strong baselines.

Related papers

• Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization [88.5582111768376]
  We study the optimization of a Transformer composed of a self-attention layer with softmax followed by a fully connected layer under gradient descent on a certain data distribution model. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noise ratio in the data model.
  arXiv  Detail & Related papers  (2024-09-28T13:24:11Z)
• Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks [93.38375271826202]
  We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. We first build a simulator by integrating Gaussian splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.
  arXiv  Detail & Related papers  (2024-06-21T13:48:37Z)
• Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning [15.792914346054502]
  We tackle the challenge of sim-to-real transfer of reinforcement learning (RL) agents for coverage path planning ( CPP) We bridge the sim-to-real gap through a semi-virtual environment, including a real robot and real-time aspects, while utilizing a simulated sensor and obstacles. We find that a high inference frequency allows first-order Markovian policies to transfer directly from simulation, while higher-order policies can be fine-tuned to further reduce the sim-to-real gap.
  arXiv  Detail & Related papers  (2024-06-07T13:24:19Z)
• Transfer Learning Study of Motion Transformer-based Trajectory Predictions [1.2972104025246092]
  Trajectory planning in autonomous driving is highly dependent on predicting the emergent behavior of other road users. Learning-based methods are currently showing impressive results in simulation-based challenges. The study aims to provide insights into possible trade-offs between computational time and performance to support effective transfers into the real world.
  arXiv  Detail & Related papers  (2024-04-12T06:50:32Z)
• COLA: Cross-city Mobility Transformer for Human Trajectory Simulation [44.157114416533915]
  We develop a Cross-city mObiLity trAnsformer (COLA) with a dedicated model-agnostic transfer framework. COLA divides the Transformer into the private modules for city-specific characteristics and the shared modules for city-universal mobility patterns. Our implemented cross-city baselines have demonstrated its superiority and effectiveness.
  arXiv  Detail & Related papers  (2024-03-04T07:45:29Z)
• Emergent Agentic Transformer from Chain of Hindsight Experience [96.56164427726203]
  We show that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches. This is the first time that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches.
  arXiv  Detail & Related papers  (2023-05-26T00:43:02Z)
• On Transforming Reinforcement Learning by Transformer: The Development Trajectory [97.79247023389445]
  Transformer, originally devised for natural language processing, has also attested significant success in computer vision. We group existing developments in two categories: architecture enhancement and trajectory optimization. We examine the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving.
  arXiv  Detail & Related papers  (2022-12-29T03:15:59Z)
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers [14.509254362627576]
  We propose to address quadrupedal locomotion tasks using Reinforcement Learning (RL) We introduce LocoTransformer, an end-to-end RL method for quadrupedal locomotion.
  arXiv  Detail & Related papers  (2021-07-08T17:41:55Z)
• Scalable Transformers for Neural Machine Translation [86.4530299266897]
  Transformer has been widely adopted in Neural Machine Translation (NMT) because of its large capacity and parallel training of sequence generation. We propose a novel scalable Transformers, which naturally contains sub-Transformers of different scales and have shared parameters. A three-stage training scheme is proposed to tackle the difficulty of training the scalable Transformers.
  arXiv  Detail & Related papers  (2021-06-04T04:04:10Z)
• Applying the Transformer to Character-level Transduction [68.91664610425114]
  The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks. We show that with a large enough batch size, the transformer does indeed outperform recurrent models for character-level tasks.
  arXiv  Detail & Related papers  (2020-05-20T17:25:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.