Monocular Robot Navigation with Self-Supervised Pretrained Vision Transformers

• URL: http://arxiv.org/abs/2203.03682v1
• Date: Mon, 7 Mar 2022 19:47:52 GMT
• Title: Monocular Robot Navigation with Self-Supervised Pretrained Vision Transformers
• Authors: Miguel Saavedra-Ruiz, Sacha Morin and Liam Paull
• Abstract summary: We train a coarse image segmentation model for the Duckietown environment using 70 training images. Our model performs coarse image segmentation at the 8x8 patch level, and the inference resolution can be adjusted to balance prediction granularity and real-time perception constraints. The resulting perception model is used as the backbone for a simple yet robust visual servoing agent.
• Score: 10.452316044889177
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we consider the problem of learning a perception model for monocular robot navigation using few annotated images. Using a Vision Transformer (ViT) pretrained with a label-free self-supervised method, we successfully train a coarse image segmentation model for the Duckietown environment using 70 training images. Our model performs coarse image segmentation at the 8x8 patch level, and the inference resolution can be adjusted to balance prediction granularity and real-time perception constraints. We study how best to adapt a ViT to our task and environment, and find that some lightweight architectures can yield good single-image segmentations at a usable frame rate, even on CPU. The resulting perception model is used as the backbone for a simple yet robust visual servoing agent, which we deploy on a differential drive mobile robot to perform two tasks: lane following and obstacle avoidance.

Related papers

• LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning [50.99807031490589]
  We introduce LLARVA, a model trained with a novel instruction tuning method to unify a range of robotic learning tasks, scenarios, and environments. We generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model. Experiments yield strong performance, demonstrating that LLARVA performs well compared to several contemporary baselines.
  arXiv  Detail & Related papers  (2024-06-17T17:55:29Z)
• Scaling Manipulation Learning with Visual Kinematic Chain Prediction [32.99644520625179]
  We propose the visual kinematics chain as a precise and universal representation of quasi-static actions for robot learning over diverse environments. We demonstrate the superior performance of VKT over BC transformers as a general agent on Calvin, RLBench, Open-X, and real robot manipulation tasks.
  arXiv  Detail & Related papers  (2024-06-12T03:10:27Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• MULLER: Multilayer Laplacian Resizer for Vision [16.67232499096539]
  We present an extremely lightweight multilayer Laplacian resizer with only a handful of trainable parameters, dubbed MULLER resizer. We show that MULLER can be easily plugged into various training pipelines, and it effectively boosts the performance of the underlying vision task with little to no extra cost.
  arXiv  Detail & Related papers  (2023-04-06T04:39:21Z)
• Real-World Robot Learning with Masked Visual Pre-training [161.88981509645416]
  In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. We find that our encoder consistently outperforms CLIP (up to 75%), supervised ImageNet pre-training (up to 81%), and training from scratch (up to 81%).
  arXiv  Detail & Related papers  (2022-10-06T17:59:01Z)
• Pro-tuning: Unified Prompt Tuning for Vision Tasks [133.12978197265596]
  Fine-tuning is the de-facto approach to leverage pre-trained vision models to perform downstream tasks. In this work, we propose parameter-efficient Prompt tuning (Pro-tuning) to adapt frozen vision models to various downstream vision tasks.
  arXiv  Detail & Related papers  (2022-07-28T21:09:31Z)
• Masked World Models for Visual Control [90.13638482124567]
  We introduce a visual model-based RL framework that decouples visual representation learning and dynamics learning. We demonstrate that our approach achieves state-of-the-art performance on a variety of visual robotic tasks.
  arXiv  Detail & Related papers  (2022-06-28T18:42:27Z)
• CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data [11.92904350972493]
  Vision-based segmentation of the robotic tool during robot-assisted surgery enables downstream applications, such as augmented reality feedback. With the introduction of deep learning, many methods were presented to solve instrument segmentation directly and solely from images. We present CaRTS, a causality-driven robot tool segmentation algorithm, that is designed based on a complementary causal model of the robot tool segmentation task.
  arXiv  Detail & Related papers  (2022-03-15T22:26:19Z)
• Masked Visual Pre-training for Motor Control [118.18189211080225]
  Self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels. We freeze the visual encoder and train neural network controllers on top with reinforcement learning. This is the first self-supervised model to exploit real-world images at scale for motor control.
  arXiv  Detail & Related papers  (2022-03-11T18:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.