Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves

• URL: http://arxiv.org/abs/2603.20850v1
• Date: Sat, 21 Mar 2026 15:12:57 GMT
• Title: Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves
• Abstract summary: Glove2Hand is a framework that translates multi-modal sensing glove HOI videos into bare hands.<n>HandSense is the first multi-modal HOI dataset featuring glove-to-hand videos with synchronized tactile and IMU signals.
• Score: 29.4768155708928
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding hand-object interaction (HOI) is fundamental to computer vision, robotics, and AR/VR. However, conventional hand videos often lack essential physical information such as contact forces and motion signals, and are prone to frequent occlusions. To address the challenges, we present Glove2Hand, a framework that translates multi-modal sensing glove HOI videos into photorealistic bare hands, while faithfully preserving the underlying physical interaction dynamics. We introduce a novel 3D Gaussian hand model that ensures temporal rendering consistency. The rendered hand is seamlessly integrated into the scene using a diffusion-based hand restorer, which effectively handles complex hand-object interactions and non-rigid deformations. Leveraging Glove2Hand, we create HandSense, the first multi-modal HOI dataset featuring glove-to-hand videos with synchronized tactile and IMU signals. We demonstrate that HandSense significantly enhances downstream bare-hand applications, including video-based contact estimation and hand tracking under severe occlusion.

Related papers

• EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset [3.8731899020772165]
  We propose a framework for event-based 3D hand mesh reconstruction from an egocentric viewpoint.<n>The proposed method incorporates a Hand Detector that estimates instance-level bounding boxes and masks for both the left and right hands.<n>We also introduce Adaptive Attention, which dynamically gates the attention based on these detection results.
  arXiv  Detail & Related papers  (2026-09-15T13:46:02Z)
• The N2D Haptic Glove: A Multi-Finger Glove for 2D Directional Force Feedback for Contact Rich Manipulation [11.79647885994712]
  Humans rely on directional fingertip forces to probe and regulate contact during manipulation.<n>Most wearable haptic gloves render only vibration or single-axis force, leaving force direction ambiguous.<n>We present the N2D Haptic Glove, a multi-finger wearable device that renders planar flexion-extension fingertip forces using capstan-drive transmissions for high-transparency force feedback.
  arXiv  Detail & Related papers  (2026-06-12T03:59:33Z)
• AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Gloves [10.234494805527943]
  We present the first systematic evaluation of vision-based hand tracking models on gloved hands.<n>Existing bare-hand models suffer from substantial performance degradation due to large appearance gap between bare-hand and glove designs.<n>We propose AirGlove, which leverages existing gloves to generalize the learned glove representations towards new gloves with limited data.
  arXiv  Detail & Related papers  (2026-02-05T00:24:02Z)
• SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation [64.3409486422946]
  We present SpriteHand, an autoregressive video generation framework for real-time synthesis of hand-object interaction videos.<n>Our model employs a causal inference architecture for autoregressive generation and leverages a hybrid post-training approach to enhance visual realism and temporal coherence.<n> Experiments demonstrate superior visual quality, physical plausibility, and interaction fidelity compared to both generative and engine-based baselines.
  arXiv  Detail & Related papers  (2025-12-01T18:13:40Z)
• BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects [70.20706475051347]
  BimArt is a novel generative approach for synthesizing 3D bimanual hand interactions with articulated objects.<n>We first generate distance-based contact maps conditioned on the object trajectory with an articulation-aware feature representation.<n>The learned contact prior is then used to guide our hand motion generator, producing diverse and realistic bimanual motions for object movement and articulation.
  arXiv  Detail & Related papers  (2024-12-06T14:23:56Z)
• FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation [11.843140646170458]
  We present FoundHand, a large-scale domain-specific diffusion model for single and dual hand images.<n>We use FoundHand-10M, a large-scale hand dataset with 2D keypoints and segmentation mask annotations.<n>Our model exhibits core capabilities that include the ability to repose hands, transfer hand appearance, and even synthesize novel views.
  arXiv  Detail & Related papers  (2024-12-03T18:58:19Z)
• HMP: Hand Motion Priors for Pose and Shape Estimation from Video [52.39020275278984]
  We develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets.
  arXiv  Detail & Related papers  (2023-12-27T22:35:33Z)
• 3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera [59.846927201816776]
  This paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions.
  arXiv  Detail & Related papers  (2023-12-21T18:59:57Z)
• GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency [57.9920824261925]
  Hands are dexterous and highly versatile manipulators that are central to how humans interact with objects and their environment. modeling realistic hand-object interactions is critical for applications in computer graphics, computer vision, and mixed reality. GRIP is a learning-based method that takes as input the 3D motion of the body and the object, and synthesizes realistic motion for both hands before, during, and after object interaction.
  arXiv  Detail & Related papers  (2023-08-22T17:59:51Z)
• RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch [28.096602057231035]
  In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass.
  arXiv  Detail & Related papers  (2023-02-28T07:37:53Z)
• 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal [85.30756038989057]
  Estimating 3D interacting hand pose from a single RGB image is essential for understanding human actions. We propose to decompose the challenging interacting hand pose estimation task and estimate the pose of each hand separately. Experiments show that the proposed method significantly outperforms previous state-of-the-art interacting hand pose estimation approaches.
  arXiv  Detail & Related papers  (2022-07-22T13:04:06Z)
• Egocentric View Hand Action Recognition by Leveraging Hand Surface and Hand Grasp Type [15.878905144552204]
  The framework synthesizes the mean curvature of the hand mesh model to encode the hand surface geometry in 3D space. Using hand grasp type and mean curvature of hand increases the performance of the hand action recognition.
  arXiv  Detail & Related papers  (2021-09-08T17:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.