Grasp Like Humans: Learning Generalizable Multi-Fingered Grasping from Human Proprioceptive Sensorimotor Integration

• URL: http://arxiv.org/abs/2509.08354v1
• Date: Wed, 10 Sep 2025 07:44:12 GMT
• Title: Grasp Like Humans: Learning Generalizable Multi-Fingered Grasping from Human Proprioceptive Sensorimotor Integration
• Authors: Ce Guo, Xieyuanli Chen, Zhiwen Zeng, Zirui Guo, Yihong Li, Haoran Xiao, Dewen Hu, Huimin Lu,
• Abstract summary: Tactile and kinesthetic perceptions are crucial for human dexterous manipulation, enabling reliable grasping of objects via sensorimotor integration.<n>We propose a novel glove-mediated tactile-kinematic perception-prediction framework for grasp skill transfer from human intuitive and natural operation to robotic execution based on imitation learning.
• Score: 26.351720551267846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tactile and kinesthetic perceptions are crucial for human dexterous manipulation, enabling reliable grasping of objects via proprioceptive sensorimotor integration. For robotic hands, even though acquiring such tactile and kinesthetic feedback is feasible, establishing a direct mapping from this sensory feedback to motor actions remains challenging. In this paper, we propose a novel glove-mediated tactile-kinematic perception-prediction framework for grasp skill transfer from human intuitive and natural operation to robotic execution based on imitation learning, and its effectiveness is validated through generalized grasping tasks, including those involving deformable objects. Firstly, we integrate a data glove to capture tactile and kinesthetic data at the joint level. The glove is adaptable for both human and robotic hands, allowing data collection from natural human hand demonstrations across different scenarios. It ensures consistency in the raw data format, enabling evaluation of grasping for both human and robotic hands. Secondly, we establish a unified representation of multi-modal inputs based on graph structures with polar coordinates. We explicitly integrate the morphological differences into the designed representation, enhancing the compatibility across different demonstrators and robotic hands. Furthermore, we introduce the Tactile-Kinesthetic Spatio-Temporal Graph Networks (TK-STGN), which leverage multidimensional subgraph convolutions and attention-based LSTM layers to extract spatio-temporal features from graph inputs to predict node-based states for each hand joint. These predictions are then mapped to final commands through a force-position hybrid mapping.

Related papers

• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction [93.88239833545623]
  We present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset.<n>We show that tactile signals provide a compact yet powerful cue for grasp understanding.<n>We aim to advance multimodal egocentric perception, embodied learning, and contact-rich robotic manipulation.
  arXiv  Detail & Related papers  (2025-12-18T18:18:17Z)
• DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation [25.208854363099352]
  This dataset contains 7,000 hours of dexterous hand-object interactions seeded from 70 hours of real human demonstrations.<n>Each entry combines synchronized multi-view RGB-D, high-precision mocap with MANO hand parameters, and per-frame contact points with physically consistent force profiles.<n>Our real-to-sim pipeline uses reinforcement learning to train policies that control an actuated MANO hand in physics simulation.
  arXiv  Detail & Related papers  (2025-10-17T16:08:14Z)
• Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing [15.970078821894758]
  We introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Robot Synesthesia is a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia.
  arXiv  Detail & Related papers  (2023-12-04T12:35:43Z)
• InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint [67.6297384588837]
  We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. We demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model.
  arXiv  Detail & Related papers  (2023-11-27T14:32:33Z)
• Tactile-Filter: Interactive Tactile Perception for Part Mating [54.46221808805662]
  Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. vision-based tactile sensors are being widely used for various robotic perception and control tasks. We present a method for interactive perception using vision-based tactile sensors for a part mating task.
  arXiv  Detail & Related papers  (2023-03-10T16:27:37Z)
• Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps [100.72245315180433]
  We present a reconfigurable data glove design to capture different modes of human hand-object interactions. The glove operates in three modes for various downstream tasks with distinct features. We evaluate the system's three modes by (i) recording hand gestures and associated forces, (ii) improving manipulation fluency in VR, and (iii) producing realistic simulation effects of various tool uses.
  arXiv  Detail & Related papers  (2023-01-14T05:35:50Z)
• Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis [6.517935794312337]
  We present an eye-in-hand learning-based approach for hand pre-shape classification from RGB sequences. We tackle the peculiarity of the eye-in-hand setting by means of a model for the human arm trajectories.
  arXiv  Detail & Related papers  (2022-03-18T09:16:48Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• From Hand-Perspective Visual Information to Grasp Type Probabilities: Deep Learning via Ranking Labels [6.772076545800592]
  We build a novel probabilistic classifier according to the Plackett-Luce model to predict the probability distribution over grasps. We indicate that the proposed model is applicable to the most popular and productive convolutional neural network frameworks.
  arXiv  Detail & Related papers  (2021-03-08T16:12:38Z)
• HANDS: A Multimodal Dataset for Modeling Towards Human Grasp Intent Inference in Prosthetic Hands [3.7886097009023376]
  Advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user. multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control.
  arXiv  Detail & Related papers  (2021-03-08T15:51:03Z)
• TactileSGNet: A Spiking Graph Neural Network for Event-based Tactile Object Recognition [17.37142241982902]
  New advances in flexible, event-driven, electronic skins may soon endow robots with touch perception capabilities similar to humans. These unique features may render current deep learning approaches such as convolutional feature extractors unsuitable for tactile learning. We propose a novel spiking graph neural network for event-based tactile object recognition.
  arXiv  Detail & Related papers  (2020-08-01T03:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.