Visual-Tactile Multimodality for Following Deformable Linear Objects
Using Reinforcement Learning
- URL: http://arxiv.org/abs/2204.00117v1
- Date: Thu, 31 Mar 2022 21:59:08 GMT
- Title: Visual-Tactile Multimodality for Following Deformable Linear Objects
Using Reinforcement Learning
- Authors: Leszek Pecyna, Siyuan Dong, Shan Luo
- Abstract summary: We study the problem of using vision and tactile inputs together to complete the task of following deformable linear objects.
We create a Reinforcement Learning agent using different sensing modalities and investigate how its behaviour can be boosted.
Our experiments show that the use of both vision and tactile inputs, together with proprioception, allows the agent to complete the task in up to 92% of cases.
- Score: 15.758583731036007
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Manipulation of deformable objects is a challenging task for a robot. It will
be problematic to use a single sensory input to track the behaviour of such
objects: vision can be subjected to occlusions, whereas tactile inputs cannot
capture the global information that is useful for the task. In this paper, we
study the problem of using vision and tactile inputs together to complete the
task of following deformable linear objects, for the first time. We create a
Reinforcement Learning agent using different sensing modalities and investigate
how its behaviour can be boosted using visual-tactile fusion, compared to using
a single sensing modality. To this end, we developed a benchmark in simulation
for manipulating the deformable linear objects using multimodal sensing inputs.
The policy of the agent uses distilled information, e.g., the pose of the
object in both visual and tactile perspectives, instead of the raw sensing
signals, so that it can be directly transferred to real environments. In this
way, we disentangle the perception system and the learned control policy. Our
extensive experiments show that the use of both vision and tactile inputs,
together with proprioception, allows the agent to complete the task in up to
92% of cases, compared to 77% when only one of the signals is given. Our
results can provide valuable insights for the future design of tactile sensors
and for deformable objects manipulation.
Related papers
- Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning [9.178588671620963]
This work aims to recognise the latent unobservable object characteristics.
vision is commonly used for object recognition by robots, but it is ineffective for detecting hidden objects.
We propose a cross-modal transfer learning approach from vision to haptic-audio.
arXiv Detail & Related papers (2024-03-15T21:18:14Z) - Neural feels with neural fields: Visuo-tactile perception for in-hand
manipulation [57.60490773016364]
We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation.
Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem.
Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation.
arXiv Detail & Related papers (2023-12-20T22:36:37Z) - What Makes Pre-Trained Visual Representations Successful for Robust
Manipulation? [57.92924256181857]
We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture.
We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
arXiv Detail & Related papers (2023-11-03T18:09:08Z) - General In-Hand Object Rotation with Vision and Touch [46.871539289388615]
We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes.
We distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs.
arXiv Detail & Related papers (2023-09-18T17:59:25Z) - Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks.
vision-based tactile sensors are being widely used for various robotic perception and control tasks.
We present a method for interactive perception using vision-based tactile sensors for a part mating task.
arXiv Detail & Related papers (2023-03-10T16:27:37Z) - Touch and Go: Learning from Human-Collected Vision and Touch [16.139106833276]
We propose a dataset with paired visual and tactile data called Touch and Go.
Human data collectors probe objects in natural environments using tactile sensors.
Our dataset spans a large number of "in the wild" objects and scenes.
arXiv Detail & Related papers (2022-11-22T18:59:32Z) - Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects.
We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model.
This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
arXiv Detail & Related papers (2021-09-09T16:04:14Z) - Learning Intuitive Physics with Multimodal Generative Models [24.342994226226786]
This paper presents a perception framework that fuses visual and tactile feedback to make predictions about the expected motion of objects in dynamic scenes.
We use a novel See-Through-your-Skin (STS) sensor that provides high resolution multimodal sensing of contact surfaces.
We validate through simulated and real-world experiments in which the resting state of an object is predicted from given initial conditions.
arXiv Detail & Related papers (2021-01-12T12:55:53Z) - Visual Imitation Made Easy [102.36509665008732]
We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots.
We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector.
We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
arXiv Detail & Related papers (2020-08-11T17:58:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.