TransHands: Repurposing Human Pose Encoders as Hand Pose Encoders

• URL: http://arxiv.org/abs/2608.22341v1
• Date: Sun, 23 Aug 2026 10:17:09 GMT
• Title: TransHands: Repurposing Human Pose Encoders as Hand Pose Encoders
• Abstract summary: We introduce TransHands, a backbone-agnostic transfer learning framework that enables pre-trained human motion encoders to be effectively adapted for 3D hand pose estimation.<n>We evaluate TransHands across four state-of-the-art motion modeling architectures, including transformer-based, graph-based, and frequency- domain models.
• Score: 3.75886080255807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lifting 3D hand poses from 2D monocular representations remains challenging due to the limited availability of large-scale, diverse 3D-annotated hand datasets, in contrast to the abundance of human body motion data. We address this limitation by transferring motion representations learned from large body pose corpora to the hand domain. We introduce TransHands, a backbone-agnostic transfer learning framework that enables pre-trained human motion encoders to be effectively adapted for 3D hand pose estimation from 2D pose inputs. Rather than training hand-specific biomechanical models from scratch, TransHands combines a two-stage training and fine-tuning strategy with a lightweight hand-specific input adaptation module that aligns hand kinematics with the representation space learned for full-body motion. We evaluate TransHands across four state-of-the-art motion modeling architectures, including transformer-based, graph-based, and frequency- domain models. Results demonstrate that motion priors learned from body pose data transfer consistently across architectures, yielding consistent accuracy gains, strong cross-domain generalization, particularly in challenging egocentric settings, and applicability for downstream tasks in real-world contexts.

Related papers

• BiTDiff: Fine-Grained 3D Conducting Motion Generation via BiMamba-Transformer Diffusion [49.503279511274435]
  BiTDiff is a novel framework for 3D conducting motion generation built upon a BiMamba-Transformer hybrid model architecture.<n>BiTDiff achieves state-of-the-art (SOTA) performance for 3D conducting motion generation on the CM-Data dataset.
  arXiv  Detail & Related papers  (2026-04-06T03:49:36Z)
• Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation [3.0979829079231447]
  We show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation.<n>We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space.<n>Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy.
  arXiv  Detail & Related papers  (2026-03-18T06:18:46Z)
• FUSION: Full-Body Unified Motion Prior for Body and Hands via Diffusion [49.026972478098266]
  Hands are central to interacting with our surroundings and conveying gestures.<n>Existing human motion synthesis methods fall short.<n>Key obstacle is the lack of large-scale datasets that jointly capture diverse full-body motion.
  arXiv  Detail & Related papers  (2026-01-07T14:18:59Z)
• AssemblyHands-X: Modeling 3D Hand-Body Coordination for Understanding Bimanual Human Activities [27.634829042887358]
  We present AssemblyHands-X, the first markerless 3D hand-body benchmark for bimanual activities.<n>Our approach combines multi-view triangulation with SMPL-X mesh fitting, yielding reliable 3D registration of hands and upper body.<n>Our experiments show pose-based action inference is more efficient and accurate than video baselines.
  arXiv  Detail & Related papers  (2025-09-28T13:52:14Z)
• CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects [14.230098033626744]
  Whole-body manipulation of articulated objects is a critical yet challenging task with broad applications in virtual humans and robotics.<n>We propose a novel coordinated diffusion noise optimization framework to achieve realistic whole-body motion.<n>We conduct extensive experiments demonstrating that our method outperforms existing approaches in motion quality and physical plausibility.
  arXiv  Detail & Related papers  (2025-05-27T17:11:50Z)
• Mocap-2-to-3: Multi-view Lifting for Monocular Motion Recovery with 2D Pretraining [49.223455189395025]
  Mocap-2-to-3 is a novel framework that performs multi-view lifting from monocular input.<n>To leverage abundant 2D data, we decompose complex 3D motion into multi-view syntheses.<n>Our method surpasses state-of-the-art approaches in both camera-space motion realism and world-grounded human positioning.
  arXiv  Detail & Related papers  (2025-03-05T06:32:49Z)
• Learning Human Motion from Monocular Videos via Cross-Modal Manifold Alignment [45.74813582690906]
  Learning 3D human motion from 2D inputs is a fundamental task in the realms of computer vision and computer graphics. We present the Video-to-Motion Generator (VTM), which leverages motion priors through cross-modal latent feature space alignment. The VTM showcases state-of-the-art performance in reconstructing 3D human motion from monocular videos.
  arXiv  Detail & Related papers  (2024-04-15T06:38:09Z)
• Realistic Human Motion Generation with Cross-Diffusion Models [30.854425772128568]
  Cross Human Motion Diffusion Model (CrossDiff) Method integrates 3D and 2D information using a shared transformer network within the training of the diffusion model. CrossDiff effectively combines the strengths of both representations to generate more realistic motion sequences.
  arXiv  Detail & Related papers  (2023-12-18T07:44:40Z)
• BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics [50.88842027976421]
  We propose BOTH57M, a novel multi-modal dataset for two-hand motion generation. Our dataset includes accurate motion tracking for the human body and hands. We also provide a strong baseline method, BOTH2Hands, for the novel task.
  arXiv  Detail & Related papers  (2023-12-13T07:30:19Z)
• MotionBERT: A Unified Perspective on Learning Human Motion Representations [46.67364057245364]
  We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. We propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. We implement motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network.
  arXiv  Detail & Related papers  (2022-10-12T19:46:25Z)
• Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics [87.17505994436308]
  We build upon the insight that body motion and hand gestures are strongly correlated in non-verbal communication settings. We formulate the learning of this prior as a prediction task of 3D hand shape over time given body motion input alone. Our hand prediction model produces convincing 3D hand gestures given only the 3D motion of the speaker's arms as input.
  arXiv  Detail & Related papers  (2020-07-23T22:58:15Z)
• Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis [72.34794624243281]
  We propose a self-supervised learning framework to disentangle variations from unlabeled video frames. Our differentiable formalization, bridging the representation gap between the 3D pose and spatial part maps, allows us to operate on videos with diverse camera movements.
  arXiv  Detail & Related papers  (2020-04-09T07:55:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.