Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins

• URL: http://arxiv.org/abs/2410.17246v2
• Date: Sat, 26 Oct 2024 02:25:50 GMT
• Title: Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins
• Authors: Venkatesh Pattabiraman, Yifeng Cao, Siddhant Haldar, Lerrel Pinto, Raunaq Bhirangi,
• Abstract summary: We present the Visuo-Skin (ViSk) framework, a simple approach that uses a transformer-based policy and treats skin sensor data as additional tokens alongside visual information. ViSk significantly outperforms both vision-only and optical tactile sensing based policies. Further analysis reveals that combining tactile and visual modalities enhances policy performance and spatial generalization, achieving an average improvement of 27.5% across tasks.
• Score: 17.412763585521688
• License:
• Abstract: While visuomotor policy learning has advanced robotic manipulation, precisely executing contact-rich tasks remains challenging due to the limitations of vision in reasoning about physical interactions. To address this, recent work has sought to integrate tactile sensing into policy learning. However, many existing approaches rely on optical tactile sensors that are either restricted to recognition tasks or require complex dimensionality reduction steps for policy learning. In this work, we explore learning policies with magnetic skin sensors, which are inherently low-dimensional, highly sensitive, and inexpensive to integrate with robotic platforms. To leverage these sensors effectively, we present the Visuo-Skin (ViSk) framework, a simple approach that uses a transformer-based policy and treats skin sensor data as additional tokens alongside visual information. Evaluated on four complex real-world tasks involving credit card swiping, plug insertion, USB insertion, and bookshelf retrieval, ViSk significantly outperforms both vision-only and optical tactile sensing based policies. Further analysis reveals that combining tactile and visual modalities enhances policy performance and spatial generalization, achieving an average improvement of 27.5% across tasks. https://visuoskin.github.io/

Related papers

• 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing [18.189782619503074]
  This paper introduces textbf3D-ViTac, a multi-modal sensing and learning system for robots. Our system features tactile sensors equipped with dense sensing units, each covering an area of 3$mm2$. We show that even low-cost robots can perform precise manipulations and significantly outperform vision-only policies.
  arXiv  Detail & Related papers  (2024-10-31T16:22:53Z)
• Learning Visuotactile Skills with Two Multifingered Hands [80.99370364907278]
  We explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data.
  arXiv  Detail & Related papers  (2024-04-25T17:59:41Z)
• DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity [12.508332341279177]
  We introduce a multi-finger robot system designed to search for and manipulate objects using the sense of touch. To achieve this, binary tactile sensors are implemented on one side of the robot hand to minimize the Sim2Real gap. We demonstrate that object search and manipulation using tactile sensors is possible even in an environment without vision information.
  arXiv  Detail & Related papers  (2024-01-23T05:37:32Z)
• The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning [60.91637862768949]
  We propose Masked Multimodal Learning (M3L) to fuse visual and tactile information in a reinforcement learning setting. M3L learns a policy and visual-tactile representations based on masked autoencoding. We evaluate M3L on three simulated environments with both visual and tactile observations.
  arXiv  Detail & Related papers  (2023-11-02T01:33:00Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• Visual-Tactile Multimodality for Following Deformable Linear Objects Using Reinforcement Learning [15.758583731036007]
  We study the problem of using vision and tactile inputs together to complete the task of following deformable linear objects. We create a Reinforcement Learning agent using different sensing modalities and investigate how its behaviour can be boosted. Our experiments show that the use of both vision and tactile inputs, together with proprioception, allows the agent to complete the task in up to 92% of cases.
  arXiv  Detail & Related papers  (2022-03-31T21:59:08Z)
• Learning to Detect Slip with Barometric Tactile Sensors and a Temporal Convolutional Neural Network [7.346580429118843]
  We present a learning-based method to detect slip using barometric tactile sensors. We train a temporal convolution neural network to detect slip, achieving high detection accuracies. We argue that barometric tactile sensing technology, combined with data-driven learning, is suitable for many manipulation tasks such as slip compensation.
  arXiv  Detail & Related papers  (2022-02-19T08:21:56Z)
• Bayesian Imitation Learning for End-to-End Mobile Manipulation [80.47771322489422]
  Augmenting policies with additional sensor inputs, such as RGB + depth cameras, is a straightforward approach to improving robot perception capabilities. We show that using the Variational Information Bottleneck to regularize convolutional neural networks improves generalization to held-out domains. We demonstrate that our method is able to help close the sim-to-real gap and successfully fuse RGB and depth modalities.
  arXiv  Detail & Related papers  (2022-02-15T17:38:30Z)
• Elastic Tactile Simulation Towards Tactile-Visual Perception [58.44106915440858]
  We propose Elastic Interaction of Particles (EIP) for tactile simulation. EIP models the tactile sensor as a group of coordinated particles, and the elastic property is applied to regulate the deformation of particles during contact. We further propose a tactile-visual perception network that enables information fusion between tactile data and visual images.
  arXiv  Detail & Related papers  (2021-08-11T03:49:59Z)
• Optical Tactile Sim-to-Real Policy Transfer via Real-to-Sim Tactile Image Translation [21.82940445333913]
  We present a suite of simulated environments tailored towards tactile robotics and reinforcement learning. A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images. This policy is implemented within a real-time control loop on a physical robot to demonstrate zero-shot sim-to-real policy transfer.
  arXiv  Detail & Related papers  (2021-06-16T13:58:35Z)
• OmniTact: A Multi-Directional High Resolution Touch Sensor [109.28703530853542]
  Existing tactile sensors are either flat, have small sensitive fields or only provide low-resolution signals. We introduce OmniTact, a multi-directional high-resolution tactile sensor. We evaluate the capabilities of OmniTact on a challenging robotic control task.
  arXiv  Detail & Related papers  (2020-03-16T01:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.