DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity
- URL: http://arxiv.org/abs/2401.12496v1
- Date: Tue, 23 Jan 2024 05:37:32 GMT
- Title: DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity
- Authors: Kang-Won Lee, Yuzhe Qin, Xiaolong Wang and Soo-Chul Lim
- Abstract summary: We introduce a multi-finger robot system designed to search for and manipulate objects using the sense of touch.
To achieve this, binary tactile sensors are implemented on one side of the robot hand to minimize the Sim2Real gap.
We demonstrate that object search and manipulation using tactile sensors is possible even in an environment without vision information.
- Score: 12.508332341279177
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The sense of touch is an essential ability for skillfully performing a
variety of tasks, providing the capacity to search and manipulate objects
without relying on visual information. Extensive research has been conducted
over time to apply these human tactile abilities to robots. In this paper, we
introduce a multi-finger robot system designed to search for and manipulate
objects using the sense of touch without relying on visual information.
Randomly located target objects are searched using tactile sensors, and the
objects are manipulated for tasks that mimic daily-life. The objective of the
study is to endow robots with human-like tactile capabilities. To achieve this,
binary tactile sensors are implemented on one side of the robot hand to
minimize the Sim2Real gap. Training the policy through reinforcement learning
in simulation and transferring the trained policy to the real environment, we
demonstrate that object search and manipulation using tactile sensors is
possible even in an environment without vision information. In addition, an
ablation study was conducted to analyze the effect of tactile information on
manipulative tasks. Our project page is available at
https://lee-kangwon.github.io/dextouch/
Related papers
- Visuo-Tactile Pretraining for Cable Plugging [8.187196813233362]
We investigate how we can incorporate tactile information into imitation learning platforms to improve performance on complex tasks.
We train a robotic agent to plug in a USB cable - a first for imitation learning.
We also explore how tactile information can be used to train non-tactile agents through a contrastive-loss pretraining process.
arXiv Detail & Related papers (2024-03-18T15:56:44Z) - Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing [16.570647733532173]
We introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation.
Robot Synesthesia is a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia.
arXiv Detail & Related papers (2023-12-04T12:35:43Z) - RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in
One-Shot [56.130215236125224]
A key challenge in robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots.
Recent research in one-shot imitation learning has shown promise in transferring trained policies to new tasks based on demonstrations.
This paper aims to unlock the potential for an agent to generalize to hundreds of real-world skills with multi-modal perception.
arXiv Detail & Related papers (2023-07-02T15:33:31Z) - Rotating without Seeing: Towards In-hand Dexterity through Touch [43.87509744768282]
We present Touch Dexterity, a new system that can perform in-hand object rotation using only touching without seeing the object.
Instead of relying on precise tactile sensing in a small region, we introduce a new system design using dense binary force sensors (touch or no touch) overlaying one side of the whole robot hand.
We train an in-hand rotation policy using Reinforcement Learning on diverse objects in simulation. Relying on touch-only sensing, we can directly deploy the policy in a real robot hand and rotate novel objects that are not presented in training.
arXiv Detail & Related papers (2023-03-20T05:38:30Z) - Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks.
vision-based tactile sensors are being widely used for various robotic perception and control tasks.
We present a method for interactive perception using vision-based tactile sensors for a part mating task.
arXiv Detail & Related papers (2023-03-10T16:27:37Z) - Open-World Object Manipulation using Pre-trained Vision-Language Models [72.87306011500084]
For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary.
We develop a simple approach, which leverages a pre-trained vision-language model to extract object-identifying information.
In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments.
arXiv Detail & Related papers (2023-03-02T01:55:10Z) - See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation [49.925499720323806]
We study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks.
We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor.
arXiv Detail & Related papers (2022-12-07T18:55:53Z) - Touch and Go: Learning from Human-Collected Vision and Touch [16.139106833276]
We propose a dataset with paired visual and tactile data called Touch and Go.
Human data collectors probe objects in natural environments using tactile sensors.
Our dataset spans a large number of "in the wild" objects and scenes.
arXiv Detail & Related papers (2022-11-22T18:59:32Z) - Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies.
The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
arXiv Detail & Related papers (2022-11-16T16:26:48Z) - Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects.
We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model.
This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
arXiv Detail & Related papers (2021-09-09T16:04:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.