See to Touch: Learning Tactile Dexterity through Visual Incentives
- URL: http://arxiv.org/abs/2309.12300v1
- Date: Thu, 21 Sep 2023 17:58:13 GMT
- Title: See to Touch: Learning Tactile Dexterity through Visual Incentives
- Authors: Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala and Lerrel Pinto
- Abstract summary: We present Tactile Adaptation from Visual Incentives (TAVI), a new framework that enhances tactile-based dexterity.
On six challenging tasks, TAVI achieves a success rate of 73% using our four-fingered Allegro robot hand.
- Score: 20.586023376454115
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Equipping multi-fingered robots with tactile sensing is crucial for achieving
the precise, contact-rich, and dexterous manipulation that humans excel at.
However, relying solely on tactile sensing fails to provide adequate cues for
reasoning about objects' spatial configurations, limiting the ability to
correct errors and adapt to changing situations. In this paper, we present
Tactile Adaptation from Visual Incentives (TAVI), a new framework that enhances
tactile-based dexterity by optimizing dexterous policies using vision-based
rewards. First, we use a contrastive-based objective to learn visual
representations. Next, we construct a reward function using these visual
representations through optimal-transport based matching on one human
demonstration. Finally, we use online reinforcement learning on our robot to
optimize tactile-based policies that maximize the visual reward. On six
challenging tasks, such as peg pick-and-place, unstacking bowls, and flipping
slender objects, TAVI achieves a success rate of 73% using our four-fingered
Allegro robot hand. The increase in performance is 108% higher than policies
using tactile and vision-based rewards and 135% higher than policies without
tactile observational input. Robot videos are best viewed on our project
website: https://see-to-touch.github.io/.
Related papers
- ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos [81.99559944822752]
We propose ViViDex to improve vision-based policy learning from human videos.
It first uses reinforcement learning with trajectory guided rewards to train state-based policies for each video.
We then rollout successful episodes from state-based policies and train a unified visual policy without using any privileged information.
arXiv Detail & Related papers (2024-04-24T07:58:28Z) - PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation [8.997347199266592]
Our goal is to equip robots with a similar capability, which we term ourmodel.
We frame this problem as the task of learning a low-dimensional visual-tactile embedding.
Using ReSkin, we collect and train PseudoTouch on a dataset comprising aligned tactile and visual data pairs.
We demonstrate the efficacy of PseudoTouch through its application to two downstream tasks: object recognition and grasp stability prediction.
arXiv Detail & Related papers (2024-03-22T10:51:31Z) - DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity [12.508332341279177]
We introduce a multi-finger robot system designed to search for and manipulate objects using the sense of touch.
To achieve this, binary tactile sensors are implemented on one side of the robot hand to minimize the Sim2Real gap.
We demonstrate that object search and manipulation using tactile sensors is possible even in an environment without vision information.
arXiv Detail & Related papers (2024-01-23T05:37:32Z) - Neural feels with neural fields: Visuo-tactile perception for in-hand
manipulation [57.60490773016364]
We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation.
Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem.
Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation.
arXiv Detail & Related papers (2023-12-20T22:36:37Z) - Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing [15.970078821894758]
We introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation.
Robot Synesthesia is a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia.
arXiv Detail & Related papers (2023-12-04T12:35:43Z) - What Matters to You? Towards Visual Representation Alignment for Robot
Learning [81.30964736676103]
When operating in service of people, robots need to optimize rewards aligned with end-user preferences.
We propose Representation-Aligned Preference-based Learning (RAPL), a method for solving the visual representation alignment problem.
arXiv Detail & Related papers (2023-10-11T23:04:07Z) - Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks.
We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders.
Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
arXiv Detail & Related papers (2023-10-04T17:59:38Z) - Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks.
vision-based tactile sensors are being widely used for various robotic perception and control tasks.
We present a method for interactive perception using vision-based tactile sensors for a part mating task.
arXiv Detail & Related papers (2023-03-10T16:27:37Z) - Learning Dexterous Grasping with Object-Centric Visual Affordances [86.49357517864937]
Dexterous robotic hands are appealing for their agility and human-like morphology.
We introduce an approach for learning dexterous grasping.
Our key idea is to embed an object-centric visual affordance model within a deep reinforcement learning loop.
arXiv Detail & Related papers (2020-09-03T04:00:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.