Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions

• URL: http://arxiv.org/abs/2509.07445v1
• Date: Tue, 09 Sep 2025 07:10:39 GMT
• Title: Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions
• Authors: Harrison Field, Max Yang, Yijiong Lin, Efi Psomopoulou, David Barton, Nathan F. Lepora,
• Abstract summary: Large language models (LLMs) are beginning to automate reward design for dexterous manipulation.<n>No prior work has considered tactile sensing, which is known to be critical for human-like dexterity.<n>We present Text2Touch, bringing LLM-crafted rewards to the challenging task of multi-axis in-hand object rotation with real-world vision based tactile sensing.
• Score: 5.590634826401321
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are beginning to automate reward design for dexterous manipulation. However, no prior work has considered tactile sensing, which is known to be critical for human-like dexterity. We present Text2Touch, bringing LLM-crafted rewards to the challenging task of multi-axis in-hand object rotation with real-world vision based tactile sensing in palm-up and palm-down configurations. Our prompt engineering strategy scales to over 70 environment variables, and sim-to-real distillation enables successful policy transfer to a tactile-enabled fully actuated four-fingered dexterous robot hand. Text2Touch significantly outperforms a carefully tuned human-engineered baseline, demonstrating superior rotation speed and stability while relying on reward functions that are an order of magnitude shorter and simpler. These results illustrate how LLM-designed rewards can significantly reduce the time from concept to deployable dexterous tactile skills, supporting more rapid and scalable multimodal robot learning. Project website: https://hpfield.github.io/text2touch-website

Related papers

• Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation [50.34179054785646]
  We present Taccel, a high-performance simulation platform that integrates IPC and ABD to model robots, tactile sensors, and objects with both accuracy and unprecedented speed.<n>Taccel provides precise physics simulation and realistic tactile signals while supporting flexible robot-sensor configurations through user-friendly APIs.<n>These capabilities position Taccel as a powerful tool for scaling up tactile robotics research and development.
  arXiv  Detail & Related papers  (2025-04-17T12:57:11Z)
• Digitizing Touch with an Artificial Multimodal Fingertip [51.7029315337739]
  Humans and robots both benefit from using touch to perceive and interact with the surrounding environment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities.
  arXiv  Detail & Related papers  (2024-11-04T18:38:50Z)
• AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch [9.606323817785114]
  We present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. Rich multi-fingered tactile sensing can detect unstable grasps and provide a reactive behavior that improves the robustness of the policy.
  arXiv  Detail & Related papers  (2024-05-12T22:51:35Z)
• Learning Visuotactile Skills with Two Multifingered Hands [80.99370364907278]
  We explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data.
  arXiv  Detail & Related papers  (2024-04-25T17:59:41Z)
• Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing [15.970078821894758]
  We introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Robot Synesthesia is a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia.
  arXiv  Detail & Related papers  (2023-12-04T12:35:43Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play [15.780086627089885]
  T-Dex is a new approach for tactile-based dexterity that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self-supervised tactile encoders. In the second phase, given a handful of demonstrations for a dexterous task, we learn non-parametric policies that combine the tactile observations with visual ones.
  arXiv  Detail & Related papers  (2023-03-21T17:59:20Z)
• Rotating without Seeing: Towards In-hand Dexterity through Touch [43.87509744768282]
  We present Touch Dexterity, a new system that can perform in-hand object rotation using only touching without seeing the object. Instead of relying on precise tactile sensing in a small region, we introduce a new system design using dense binary force sensors (touch or no touch) overlaying one side of the whole robot hand. We train an in-hand rotation policy using Reinforcement Learning on diverse objects in simulation. Relying on touch-only sensing, we can directly deploy the policy in a real robot hand and rotate novel objects that are not presented in training.
  arXiv  Detail & Related papers  (2023-03-20T05:38:30Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• PyTouch: A Machine Learning Library for Touch Processing [68.32055581488557]
  We present PyTouch, the first machine learning library dedicated to the processing of touch sensing signals. PyTouch is designed to be modular, easy-to-use and provides state-of-the-art touch processing capabilities as a service. We evaluate PyTouch on real-world data from several tactile sensors on touch processing tasks such as touch detection, slip and object pose estimations.
  arXiv  Detail & Related papers  (2021-05-26T18:55:18Z)
• TACTO: A Fast, Flexible and Open-source Simulator for High-Resolution Vision-based Tactile Sensors [8.497185333795477]
  TACTO is a fast, flexible and open-source simulator for vision-based tactile sensors. It can render realistic high-resolution touch readings at hundreds of frames per second. We demonstrate TACTO on a perceptual task, by learning to predict grasp stability using touch from 1 million grasps.
  arXiv  Detail & Related papers  (2020-12-15T17:54:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.